Tim, many of us have been through as bad or worse. I have, many posters have, my nephew has been through much worse in his divorce. I'm not saying this to sound mean, I'm saying it because we do understand and when dadtotheend gives you advice I would say to listen.
The courts can't stop people from being what they are. You get a traffic ticket, that won't stop you from speeding next time or parking in the wrong place. The courts can't enforce good sense or ethics or sanity.
You can make your case, but if you can't bring yourself to stand up and tell the truth in court about her, you have nothing to complain about with the courts. You have to focus on what your kids need, and produce the facts to get the courts to support that.
If you can't afford your house, sell it. You wouldn't be the first and you won't be the last. You aren't helping yourself by having multiple things stressing you out. Simplify your life, cut down on your problems.
Trying to take full custody away from the other parent is huge and it is ugly. Going for shared custody is reasonable. You can present yourself as an excellent parent with the ways and means to care for your children. If she is that bad a mother, if they are at risk, it is your duty to them to point it out. If she is a decent mother, then go for shared parenting. We can't make that decision for you.
In the state you seem to be in, you will go through hell trying to start any kind of legal proceeding now. Focus on your kids. Make sure you have them at least half the time. Don't let her keep them from you. You have every right to seek joint custody and shared access, 50/50. If she is a threat to them, or neglectful, show it and go for full, but make your decision and make sure you can back it up.
Focus on straightening out your life so you can provide a stable home for your children. Forget about your ex for now. Separate from her, separate your mind from her, your emotions, your finances, your time. Focus on your kids.