Worth saying out loud: user-agent is not identity. Verify AI crawlers by reverse DNS or the provider's published IP ranges - the ones worth letting in all publish them.