Rising mortgage rates and home prices are hitting the second-home market much harder than the primary-home market.