Firstly you need to appear for entrance exam for preferred. Colleges for Msc Mathematics.

Based on that after graduation you shall join the course provided you meet all eligibility criteria.

Syllabus Secondly will vary for all institute /University so cannot be generic.

For admission to Msc courses at IIT and NIT students need to appear for IIT Jam Exam and syllabus shared below for preparation.

https://university.careers360.com/articles/jam-syllabus/amp