this is the grandparents that recently lost their son ( your daughters dad right )( if not ignore the rest of the posting )

yes lateness is an issue but i think perhaps your daughter or even they may have unintentionally extended it because here is a family missing one of the dads ( aka their son ) on fathers day - and if one of the grandparents is the father of said son - may have just been trying to cling to his son somehow through your daughter- the next best thing in their eyes is one of his offspring
especially if you are not remarried and there is no father figure at home - they may have not felt it was an issue on that day to be prompt.

any chance your daughter was also having a hard time with it being fathers day and wanted to be with her dads parents but didnt want to upset you by saying so ?

any other day but fathers day i would be a bit more upset but i might let it go for fathers day you have a teen without a dad- a father without his son etc no matter what the reasons it can be turned to make you look very bad if you bring it up for this time ( esp by a strong willed teen girl during an argument - i have one of them and she will remember things i cant hardly remember if she is mad at me )and approach them for the next visit .

next year unless you have someone special you see on fathers day i would maybe let them have them for most of that day - i know you are both mom and dad to your kids - but they have lost something that just cant be replaced but your kids can make them feel a little closer to that person and better as they go thru their grief.