Maybe there is reason to be worried, but I wouldn't be so concerned.
From the response you showed us I don't necessarily see they claiming ownership over your site. They mention IP, but developers normally mean they want to be given credit for their work, not necessarily that they claim your site is theirs. Your site is yours, you pay the hosting and that counts for something. They own the IP of the work, meaning they claim the right to reuse that code in other projects and sell that to someone else. That still could be a risk for you, I get it. But your business seems pretty local, the chances of them doing the same project for a competitor of yours look slim.
And by looking at their own website, I see a small team of mostly young computer geeks. It doesn't look like they would spend much time tracking all the past clients to see if they are still behind the business or just retired and sold it to someone else, and then spend time and money pursuing those clients on the other side of the globe. Big corporations with big law departments may do that, but often not even them.
When someone buys someone else's business they normally buy it with everything in it, including the website and all the software and hardware used in the company. I'm not sure there's an issue with this.
Besides this, your initial complaint comes from the fact that they decided without you to put something in the code you do not approve. I agree with you, but it is a matter of principles more than law and this is a fuzzy area.
When you hire a developer or agency you hire them to develop features, not to tell them which code to use. This code (the author Schema.org) does not add any feature to the website nor it is necessary for it to work. But unless you told them at the start that you did not want any code that wasn't necessary for the features you asked, they did do what you asked of them. They did also something you didn't ask for, true. But you did not ask them not to. And what they did does not show for regular users and does not prevent you from doing business.
Without an NDA signed by both sides, finding a solution to your concerns may be difficult. Maybe a possible way would be to give them some more work on a new contract, and then ask them to sign an NDA that does not allow them to reuse the code for a competitor or keep a live copy of your site on the Internet after the project is done. If you let them keep the author Schema.org and reuse parts of the code on projects not related to your business, they are likely to sign it and you'll feel safer. Or they won't sign it if you parted ways in an unfriendly manner. You never know.
This is just my opinion. It's not worth much, and if you want a clearer answer, my advise is to consult a lawyer specialized in software licensing and rights. Search for one locally, don't wait for Upwork users or staff to fix it.
