Ben, many things are allowed in some quantities. The issue isn’t the studies or material themselves(unless it’s apikorsus), but rather the kevius itself is the problem. Making a set time for learning something other than Torah is the issue, not the math or science itself.
The rashba did, however, make a cherem against young people learning secular studies in his time and place – like anything else, if something is leading to problems, it can be regulated.