You are obviously being thoughtful and are anxious not to cause offence, but you are quite right too. Don’t be made to feel that you must go on another date because the shadchan says so. You can politely decline, saying the date was obviously a nice person but not for you, you could detect there was no chemistry and do not wish to lead anyone on.
Chemistry is very important, not often discussed but crucial if a couple are actually going to get along, sure you need to have things in common, but you cannot be forced to find somone attractive. Be firm but polite and defend yourself, it is harder to let someone down after a serious of dates then just one.