If you don't want to depend on an employer, you can be self-employed or become an employer yourself.
The latter two probably need more than skills, they need connections and/or capital. And even then, you can't have employers without employees, so for each employer, we probably would have at least 5 or so people that are employees.
I mean, it's not like every person that has had a criminal record can found his own company or do freelance work. I mean, if that's possible, then in theory every person in the world could be self-employed... but on average, they won't.
So you end up with a "caste" of permanently unemployed people anyway if something isn't fixed.
Just to be sure, when people say "loser pays" they only mean the plaintiff, right? Being sued, then losing, then being forced to pay extra for the money spent suing you also seems like a recipe for abuse.