u/treetopnintendo990

I was thinking about Majora's Mask and got a question I can't figure out. Link learns the "Oath To Order" from the first giant he encounters if he goes to Clock Town and plays the song on the ocarina only the giants he freed thus far show up to stop the Moon.

I don't get why not all four come though as he technically played the melody that is associated with all the giants and sure, they are imprisoned by the bosses on the temple but then how does any of them come at all when Link rescued them in different timelines? I am just so confused about this...