On wartime US track, there is a fair bit of use available after the pads wear down but ultimately, yes the link body starts to wear away and that can condemn the links. The pins themselves are pressed into the link and retained, by friction only, by the rubber rings moulded onto the pins. These can tear loose but that is incidental to the pads wearing.
Many post-war tracks have replaceable, bolt on rubber pads.