Tommy Bahama owner Oxford Industries warned consumers are pulling back on their spending and new tariffs will impact results.