Large amounts of firecrackers and raw materials had been stored in the Om Siva Shakti factory with major Hindu festivals weeks away.
Police officer Najmul Huda said rescue workers and firefighters initially could not get into the building as the fire raged, triggering deafening explosions of firecrackers.
Workers fled the area, but many were killed as they waited at a nearby warehouse which also caught fire and exploded, Huda told reporters.
The Press Trust of India news agency said that about 300 people were working in the factory and 52 died.
Huda said authorities had suspended the factory's license a day earlier after finding major safety violations. The management, however, operated the factory illegally on Wednesday, he said.
The fire, which spread to 40 of the factory's 60 rooms, was put out more than five hours after it began, witnesses and news reports said. Photographs showed the factory had burned to rubble, and fireworks littered the ground.
The cause of the fire was not immediately known, police officer P. Karupaiah said.
Sivakasi town in Tamil Nadu state is India's biggest hub for the manufacture of matches and firecrackers. The town is about 500 kilometers (310 miles) southwest of Chennai, the state capital.
The CNN-IBN television news channel said rescue workers had completed a search of the devastated building for trapped workers.
