Lews Castle was constructed in 1847 for James Matheson, who made his fortune from the opium trade. Opium exported from China and India was often mixed with tobacco by smokers.