I understand your point that "we don't need to work in all cases, just the ones that matter", however, this faces its own set of issues. First, you leave things behind, is that ethical? Second, what is good enough, how can we tell? Third, if we have to deal with unknowns of some kind but not others, is that really possible? Fourth, and most important, not all unknowns are people related, like lying or withholding. Would I survive falling from the fifth floor? Some people do, some don't. That's an unknown. Finally, the algorithm needs to work despite humans. If you need to rely on we behaving a certain way it is doomed to failure. Even if you assume they will behave morally, badly or on their best interests.

Maybe this goes against your views on philosophy, but a good thought experiment is for you to consider things without taking people into account first. Can you reason backwards on the future of nature? I am not talking about human impact, you can think of a time before humans.