X's threats and how to respond

mama bear of 2

So X recently threatened to call social services on me. I don't know what he thinks he's going to tell them, but his intention is to get the kids 100% of the time. 

My first response was to tell him to go ahead an call. I have nothing to hide and no reason to fear them. My second response was to tell him I didn't know if I'd allow him to have the kids because I don't think he will bring them back. We don't have a CO as of yet, so either of us can keep the kids from the other.

I feel like it's very important to be the most reasonable person I can, and not allow my emotions to rule my decisions. I believe my actions will show me as fair and calm and rational.

I also feel like this threat was an empty one, that he has no intention of calling SS and I think the kids should have the same pattern that they have had for the last several weeks. I absolutely do NOT want to play games with them.

I guess I'm torn between believing his threats and giving the kids time with him and I'm not really sure what to do.