Human brain (average 1100 to 1300cm^3) seem to have deteriorated at least in size, in the last couple of hundred thousands years. How does that decline translate to intelligence? Not sure but I would speculate that it probably does correlate!
As to the cause, anthropologists seem to be rather in agreement that nutrition may have had something to do with that. Humankind seemed to have regressed from very healthy nomadic lifestyle based on hunting and fishing to a settled one based on farming and consuming starchy plant based food. Which correlated with the deterioration of physical health, body size and brain size. [add refs later]
According to this research. Neanderthal did not have to use fire to cook their exclusively animal based meals. My thought is that they must also have consumed plenty of DHA from animal brains and spinal cords, together with all the fats to grow and feed their very large brains of average volume 1600 cm^3 (some sources give bigger estimates) . Ability to use fire gave Cro-Magnon (average brain volume 1500 or 1600 cm^3) an advantage when game became scarce and food versatility became useful, by the end of the last ice age.