Danny and Frozen nailed it. You are over spined on your arrows. That's why it got better when you put a heavier point on. Also the vanes have got to go! If you look it up, there's a way to check spine by shooting at a target 10' away. If the arrows kick one way they are too heavily spined, and if they kick the other they are too light. Look it up though, I forget which way is which, haha.
Have fun with that recurve! I'm getting back into trad this year as well. Hoping to get one with the longbow.