Fidgety, but not crying. Pick him up?

Temp53431437508241

He's almost 3 weeks old, and is pretty set on a 3 hour routine. He ate a couple of hours ago, and is in his rock n play. He's a noisy little dude normally. So he's quiet and sleeping, then half wakes up and fidgets, making noises, but not crying. If I don't pick him up he'll usually calm down. Should I let him do that? I'm feeling guilty sometimes.