He's fine. He's fine! He's a happy, healthy baby. He just prefers boob or food to anything that comes out of a bottle.

So yes, you should stop stressing.

Though it if is of any help, does he like yoghurts? or those baby fromage frais things? or cheese? - all ways to sneak extra dairy into his diet under the guise of solid food.