There is a fat known as casein which is present in the milk which causes the milk to take more time to cool than any other drinks. Milk is a mixture of both casein and water. When the milk is cool, the casein and water gets separated which causes casein to form in a big molecule. When casein gets formed in a giant molecule, the milk gets useless. When the milk is heated these fats also start getting heated which causes water molecules to combine with casein and form a liquid.
When milk is added to tea or coffee then the powder is also added which acts like caffeine. This caffeine contains chemical and acts as an emulsifying agent. These agents break the fat into smaller parts which helps to cool the tea or coffee faster.
Another reason can be that milk contains proteins, fat etc. The protein which is present contains amino acid molecules. These molecules are connected to peptide bonds. So, when milk is heated these bonds become stronger. In this, the melting point of saturated fatty acids depends on its boiling point. So, as we boil the milk more the time it will take to cool whereas as a powder is added to tea or coffee the fat becomes less and the milk gets cooler at room temperature.