I once had my carbon monoxide alarm go off on a Yom Tov. One of the firefighters who came (yes, I called 911 on Yom Tov for it!) told me that any time you leave a flame on, you need to leave a window open. Since then, I leave my kitchen window open, and I haven’t had any problems.
Just to note (for Yomim Tovim at least), if your oven has a pilot light (as opposed to an electric ignition), R’ Shachter holds that you can shut the oven or burner completely, because the flame isn’t being extinguished. It’s only being shrunk back down to the pilot. In such a case, for safety purposes, you should shut the burner whenever you’re not using it.