とめる and やめる are both “to stop” and even use exactly the same kanji as “止める”, but there are distinctive differences in the usage. (Note とめる and やめる have other meanings, often using other kanjis. Please refer to the links for more details)

やめる is used only for people.

• Stop habitual actions
  • 彼はタバコをやめる決意をした -> He vowed to stop, quit, smoking
  • 彼女は彼に会うのをやめた -> She stopped seeing him
  • このゲームは一度やり始めたらやめられない -> Once you start playing this game you can’t stop
• Make somebody stop doing something
  • 大音量で音楽をかけるのはやめてくれ -> Stop playing music at such loud volume
  • 二人共、喧嘩はやめなよ -> You two, stop fighting, bickering
  • 弟をからかうのはやめなさい -> Stop teasing your little brother
• Decide not to do something (before you do it)
  • 外出しようかと思ったがやめた -> I thought about going out, but decided not to
  • 買おうかと思ったがやっぱりやめた -> I thought about buying it, but decided not to
• Prevent somebody from doing something in やめさせる
  • 彼女はAVに出ようとしたが、親にやめさせられた -> She tried to do porn, but her parents stopped her

All the above CANNOT be substituted with とめる

とめる is for both people and things.

• 車をあそこにとめて下さい -> Stop the car over there
• 印刷はどうやってとめれば良い？ -> How can I stop the printing?
• 痛みをとめる薬 -> A medication to stop pain

When about people, とめる is to physically stop someone

• 彼をとめようとしたが、彼は私を押しのけて出て行った -> I tried to stop him, but he pushed me away and left

とめる can be used like やめさせる (= stop somebody from doing something), but the sentence always refers to stopping the physical action that leads to the result.

• 容疑者は首相に襲いかかろうとしたが、護衛にとめられた -> The suspect tried to attack the prime minister, but the bodyguard stopped him (physically stopped the suspect, which prevented the would-be harm)

The below are both valid sentences.

• 喧嘩をとめろ！ -> Stop the fight!
• 喧嘩をやめさせろ！ -> Make them stop fighting!