Church not including housing allowance on paycheck!

by James
(Wisconsin)

If my church hasn't been including the housing allowance deduction on my paycheck and I have paid tax on all of my income, will this ultimately affect me? I will simply receive the difference as a refund come tax season if I file correctly, yes?