Avraham Avinu came from a non-Frum family, too. His Father didn’t send him to Yeshiva, either. His “Mechutan and Daughter-in-Law’s brother (Our Uncle and Great Grandfather) were both crooks.
I prefer not to extrapolate things like this from the Avos, but if you must, you should also notice that his one son became a rasha and is the father of the Arabs, another grandson was a rasha and his family became amalek.
So, if we are going the yichus track here, I am not sure the results were too great.