What model of Kindle do you have ? Some models/generations have higher power requirement.
What does the back of your kindle say regarding power ? (500mAh?)
Is your usb port on the pc a usb 2 or 3 ?
The maximal charge on a usb 2 port is 500mAh, so if it charges on the pc it should charge on a 500mAh usb adapter too.
(If you charge it via a usb3 port, then its 900mAh! and i would recommend a usb adapter of 1000mAh)
Check that your usb wall adapter minimally matches the mAh that the kindle says on the back.
(Like Warwick says, more is not a bad thing, the kindle will take what it needs)
Some phone chargers give out less then 500mAh, check if you have one that's minimal 500mAh.
Its said that the charging will take up to 4 hours. But i guess that depends on the model too.
Is it the original usb cable ? If not, did you try a different one ?
(have seen cheap Chinese cables doing strange things)
Another tip for charging with the pc (found on the internet) :
"After you connect your kindle to the pc, click on the "Start" button, select "Computer" from the options listed, right-click on the icon for the Kindle and select "Eject" from the pop-up menu. This will cause Kindle to exit USB drive mode and bring up the Home screen. Check on the kindle if its charging.
USB drive mode uses extra power so charging takes longer, turning it off makes charging go faster."