Get your babies, and file the papers with the court. There is a flight risk, he kept the children past the agreement and refused to return them, causing upset, confusion & turmoil.
It's one thing to ensure the children have a chance to be with each parent, but when the other breaks what's been agreed & followed after this length of time, then it's time to take it seriously.
You need to request that he not be allowed to remove the children from the province without your prior written consent, if they do not already have them, make sure you also cover the passport issue. Also, you must request that the court clearly state that the police can enforce the custody order, thus if he attempts to pull the stunt again, you can involve the police.
You are primary, that has been established & the children are with you the majority of the time, look to having the order clearly state his visitation, but you will need to be child oriented and consider holiday & summer time access as well. Also, make sure that the order will cover future access for when the children will be in school and what may or may not work best for them.
Take it to court, you need to for the best of the children after this.
Good luck to you, and all the best.
EDIT: As to the costs, if need be request your own costs to be paid as well, your reasoning of course would be that you previously have an agreement, although not court ordered, that has been working fine for the children for X period of time, but due to this recent incident, you felt that for the best of the children, an emergency order was required.