How do you choose a baseline for a new image segmentation method?

I keep running into the same problem when reviewing and writing papers: the baseline choice often decides whether a new segmentation method looks good.

Some authors compare against the latest transformer models, others against older U-Net variants, and the results are hard to compare.

How do you decide which baselines are fair and necessary for a new method?