Over and over again I see a paper that is more or less as good as many papers before it, but whether it squeaks in, or gets an oral, or gets rejected, all seem to depend on luck. I have seen bad papers get in with faked data or other real faults because the reviewers were positive and inattentive.
I agree but the problem is also that faked data is incredible hard or even impossible to spot with the current system. You would need to standardize the whole process (code request, exact experiment description, code explanations, creating docker image for reproducability, computational cost, ...). Then the reviewers would need to run some of the experiments themselves aswell (alongside additional experiments to make sure you are not cherrypicking results). This would take a tremendous amount of time and resources
I agree but the problem is also that faked data is incredible hard or even impossible to spot with the current system. You would need to standardize the whole process (code request, exact experiment description, code explanations, creating docker image for reproducability, computational cost, ...). Then the reviewers would need to run some of the experiments themselves aswell (alongside additional experiments to make sure you are not cherrypicking results). This would take a tremendous amount of time and resources