T613t,
You’re misstating the issue.
Much secular knowledge is valuable and acceptable. Much is harmful and assur; the charedim have avoided even the former. The reason is that during one’s formative years, it is necessary to focus on learning in order to develop properly as a ben Torah, at the expense of what is inherently valuable.