1) For when they start holding things: this soft-sided sorter comes with three different kinds of stuff inside, hard rattly things, vinyl squeaky things, and soft fabric things, all of which are likely to be of great interest to a young child (and, judging by SteelyKid, to be tasty, too). Plus you can zip up the top and keep all the things inside, which is great for travel. (FWIW its availability seems to wax and wane, as I've bought it for cheaper than the current price and avoided buying it at other times.)
2) For when they start manipulating things: I spent way too much time looking for stacking rings, and these look and feel really nice, plus the slightly different design lets kids experiment with different stacking patterns as they grow.
(Bonus: not a toy, and don't give it as a gift unless you know the kid is using this particular kind of pacifier, but these pacifier holders were essential for us.)
And not for babies, but while I have my Amazon history open:
A really nice tea set.
Open the doors and find animals.
Magna-Tiles are expensive but they're very durable and endlessly fascinating. Good for as young as three, and possibly earlier.
(Finally, if you like to give books, try the "That's Not My..." board books from Usborne (available in any bookstore); any of Sandra Boynton's board books; any of Mo Willems' Pigeon or Elephant & Piggie books; or Peekaboo Morning.)