Those who do sit in the Succah on Shmini Atzeres are in direct violation of the how most (all?) Rishonim understand the Gemara, and should keep eating in the sukah, because they have on whom to rely and minhag is oker halachah.
I wouldn’t call following Shulchan Aruch being oker halachah. The Tur and Bais Yosef both bring the Rosh (same idea as Tosafos) but don’t make any chilluk for weather, and certainly not for how common it is to eat in a sukkah when there’s no mitzvah. The point is not taking a survey of how common it is; essentially, as Yom Tov Sheini, there’s a chiyuv to sit, snd as long as it’s conceivable to be toleh sitting in the sukkah on it being pleasant, we do so. Lulav would be muktzeh, so it’s not conceivable.