That's only a 4 second variation & I've now messaged you with something to try out.
What it comes down to is when accessing an HDD or an SSD for the first time after a reboot, the OS hasn't yet got an in-memory cache about the files & folders on the drive which means it's going to be slow until that information is cached which will make it faster the next time around things need to be accessed.
So I'd say it's normal but an HDD makes it more noticeable & your setup vs the size of the playlist & what I'm doing is triggering a worst case response which I'm hoping what I've messed will help to improve the reported situation as well as subsequent loads once the OS cache has been built.
-dro