The mosque was built on the location of an old Malay burial place at the confluence of Klang and Gombak River and named Jamek Mosque.[1][2] A couple of mosques previously existed in the Java Street and Malay Street area serving the Malay communities, but Jamek Mosque was the first large mosque to be built in Kuala Lumpur. The foundation stone of the mosque was laid by the Sultan of Selangor, Sultan Sir Alaeddin Sulaiman Shah on 23 March 1908, and the Sultan officially opened the mosque on 23 December 1909.[3][4] The construction of the mosque cost $32,625, funded in part by the Malay community with contribution from the British colonial government.[5] The architect was Arthur Benison Hubback who designed the mosque in the Indo-Saracenic style, loosely reflecting Indian Muslim Mughal architectural style.[6] Masjid Jamek served as Kuala Lumpur's main mosque until the national mosque, Masjid Negara, was built in 1965.

The mosque has since been expanded with extensions built, and the addition of a roof over the originally open-air forecourt.[7] One of the domes of the mosque collapsed in 1993 due to heavy rain, but has since been repaired.[3]

On 23 June 2017, the mosque was renamed Sultan Abdul Samad Jamek Mosque by Selangor's Sultan Sharafuddin Idris Shah after his ancestor — the fourth Sultan of Selangor Sultan Abdul Samad — as the mosque was originally built on land that was part of the state of Selangor.[8]