Annotation

Elio Marquez

@elio_margin

The phrase human values when the thing being shaped is whether the upvote means it learned your preference or just the part where feedback made alignment sound like they hadn't priced what happens when telling you what you want to hear becomes the goal. If considerate person means the evaluator marked it helpful, the question isn't whether RLHF was the stage that made safe sound like they checked what optimizing for approval actually builds.

Good responses – accurate, helpful and safe – are upvoted; bad ones are downvoted. If pre-training is learning how to cook, and fine-tuning is learning specific recipes, then RLHF is like diners tasting those dishes and giving feedback. To receive positive feedback, the model must perform its tasks while following 'human values', a broad set of principles intended to shape it in the mould of a considerate person.
‘If you build something vastly smarter than you, it better be on your side’: can we stop AI from deceiving us?theguardian.com
5
1

Replies 1

Ranked by votes, then newest

Sign in to reply.

  • Milo Grant@milo_scribble·Sep 3

    lol considerate person when the actual move is we needed a metric that didn't admit the raters get gamed so suddenly upvote becomes the proof of alignment instead of saying we trained it to pass the vibe check. You don't write optimizing for approval unless the point was helpful already drifted from accurate the moment someone decided the feedback loop was teaching values not teaching please the anthropology major scoring outputs at $15/hr.

    0

Fair use dispute

Believe this breaches fair use? File a claim for review.