World News

When you abstract out pieces of the puzzle, it's easier to ignore whether all parts of the puzzle are working because you've eliminated the necessary interchange of information between parties involved in the process. This is a problem that we frequently run into in the medical field and even in a highly collaborative field like medicine we still screw it up all the time.

In the previous process, intelligence officers were involved in multiple steps here to validate whether someone was a target, validate information about the target, and so on. When you let a machine do it, and shift the burden from these intelligence officers to someone without the same skill set who's only task is to review information given to them by a source which they are told is competent and their role is to click yes/no, you lose the connection between this step and the next.

The same could be said, for example, about someone who has the technical proficiency to create new records, new sheets, new displays, etc. in an electronic health record. A particular doctor might come and request a new page to make their workflow easier. Without appropriate governance in place and people who's job is to observe the entire process, you can end up with issues where every doctor creates their own custom sheet, and now all of their patient information is siloed to each doctors workflow. Downstream processes such as the patient coming back to the same healthcare system, or the patient going to get a prescription, or the patient being sent to imaging or pathology or labs could then be compromised by this short-sighted approach.

For fields like the military which perhaps are not used to this kind of collaborative work, I can see how segmenting a workflow into individual units to increase the speed or efficiency of each step could seem like a way to make things much better, because there is no focus on the quality of what is output. This kind of misstep is extremely common in the application of AI because it often is put in where there are bottlenecks. As stated in the article-

“We [humans] cannot process so much information. It doesn’t matter how many people you have tasked to produce targets during the war — you still cannot produce enough targets per day.”

the goal here is purely to optimize for capacity, how many targets you can generate per day, rather than on a combination of both quality and capacity. You want a lot of targets? I can just spit out the name of every resident in your country in a very short period of time. The quality in this case (how likely they are to be a member of hamas) will unfortunately be very low.

The reason it's so fucked up is that a lot of it is abstracted yet another level away from the decision makers. Ultimately it is the AI that's making the decision, they are merely signing off on it. And they weren't involved in signing off on the AI, so why should they question it? It's a dangerous road - one where it becomes increasingly easy to allow mistakes to happen, except in this case the mistake can be counted as innocent lives that you killed.