CPU and RAM are irrelevant. For a 100Mbps dedi server the TOTAL maximum speeds that you can get is 12.5MB/s
If you add your uploads from all active torrents you get just a couple hundred kbps? then you definitely have a problem. If they add up to 12MB/s then you are fine.