History of Bhaidooj can be traced back to the ancient Hindu mythology but there is no concrete information as to when custom of sister's applying tilak on brother's forehead came into existence. To back the custom, there are several interesting stories related to Bhaidooj History.
Story goes that once Yamraj, the Hindu mythological God of death visited his sister Yami on this day. Yami welcomed his brother warmly and applied tilak on his forehead. Yamraj was so happy that he declared that anyone who receives teeka from his sister on the auspicious day of Bhaidooj will not receive the fury of hell. Yamraj who ate with sister Yami on that day, was so happy that he said that brothers who eat with their sisters on Bhai Dooj will be blessed with a long and fruitful life.
Another popular story of bhai beej is related to Lord Krishna which says that after killing the demon Narakasur, Krishna visited his sister Subhadra on the day of 'dooj', (second day after full moon). Subhadra welcomed her brother traditionally by applying tilak. The custom came to be followed by other sisters and eventually became an annual festival.
An incident related to Lord Mahavira is also connected with Bhaidooj history. It says after Mahavira attained nirvana his brother Raja Nandivardhan became very depressed due to Lord's absence. Nandivardhan was consoled by his caring sister Sudarshana. Bhai Dooj thus came to be associated with a sisters caring spirit.