How do you handle class imbalance when the minority class matters most?

I work on problems where the rare class is the one we actually care about, and standard resampling or reweighting often feels arbitrary.

I am never sure how much of the reported improvement is real and how much comes from tuning on the test set.

How do you decide which imbalance strategy to use, and how do you report results honestly?