Signal does have your phone number, which is a problem.
On the other hand, the only information linked to that phone number is, “the person with this phone number uses signal”. AFAIK your phone number is not linked to your contacts, your message content, etc.
So in practice, the fact that Signal has your phone number is probably only a problem insofar as you don’t want anybody to know that you use Signal.
But to be fair, why have that issue if you don’t have to. Signal is actually good, still, but there are even better alternatives.
AFAIK the client slurps up your contacts, but the E2E encryption ensures that the Signal server cannot actually see those.