Black hard-ash residue does accumulates in the flash hole; removing it properly is an important part of the cleaning process. I made my own tools to get this job done properly. I was amazed how much "stuff" I manage to pull-out first time! So you may want to check this out.
The ignition nipple itself may be not the best option for the powder/primer combo you use; you may want to look into, & even change/upgrade your ignition nipple. (not sure which rifle/ignition nipple/black powder/primer combination you use)
fyi: I use a TC Impact loaded with BH 209 & CCI 209 primers for a couple of years now; no ignition issues so far, including during several harsh cold/wet control hunts.
