The first building recorded on the site of Castle Stalker was a small fort constructed by the Clan MacDougall around 1320AD.

Around 1388AD, the Stewards took over the lordship of Lorn from the McDougall's, and the fort fell into their possession. They erected Castle Stalker in its current form around 1440AD.

Castle Stalker subsequently changed hands a number of times, before it was abandoned in 1840AD. It fell into disrepair until being purchased by Charles Stewart in 1908AD, who carried out basic conservation work. Stewart Allward acquired the castle in 1965AD, and completed its restoration over the course of the following ten years. At the time of writing, it remains in private ownership, and is only occasionally opened to the public.