Not always,but depends on the university syllabus which the college follows (in India). If your university provides BSc Physics Only with Mathematics subsidiary options then you will have to study them both,but some universities do have options of Accountancy,Statistics ,etc in place of Mathematics and Chemistry.Check out the subsidiary subject options with various universities before opting and there may be more options i believe .

Hope,this information helpful for you.Goodluck.