Skew left braces with non-trivial annihilator