There is a massive literature on the philosophy of science and the project of constructing science from observations and logic. An overview of the issues very briefly mentioned in this chapter can be found in Peter Godfrey-Smith’s Theory and Reality (University of Chicago, 2003) and virtually any textbook on the philosophy of science.
Thomas Kuhn’s book The Structure of Scientific Revolutions (University of Chicago, 1962) has been enormously influential. There are also multiple overviews of his argument available on the internet, including a useful entry on Wikipedia.
A more radical view of scientific progress can be found in Paul Feyerabend’s Against Method (New Left Books, 1975). As his title suggests, Feyerabend argues that there is no scientific method, and our best bet is to try a very broad range of approaches to understanding the world and see what happens—“anything goes” is his slogan.