The Patriots are usually have slow starts to the season. Right now, their offense is winning the games for them. It's possible their defense will improve throughout the season (I hope not).
According to that article, it seems Gillislee and Gilmore are the new whipping boys. I wished the Bills would have kept Gillislee, but it didn't work out. With Gilmore and with any good player in the secondary, they are usually very hard to resign after their rookie deals as their second contract is usually astronomical. We've seen the Bills lose good secondary players over the years to free agency: Winfield, Clements, Byrd, and Greer, so it's no surprise the Bills did not resign Gilmore.
Another thing I noticed with Gilmore over the years with the Bills is that he seems to have struggles early on in the season, but gets progressively better as the season goes along. I'm sure the same thing is happening with him in NE, which is what people who never followed him closely are finding out about him.