That may depend on several factors: How long you've been seeing him, how comfortable you would feel being direct with him, etc. Most therapists would not take offense if you wanted to try something different.
Alternatively, you could:
- say you're taking a break
- tell him you don't want to make another appointment just now but will contact him when you do
- tell him you want to focus on other things in your life for a while