With a single gigabit port coming out of the tuner, it's either plugged into a spare NIC on the server with hard-coded IP addresses, or putting load on a switch. If it's drives, you might see extremely high disk activity. Maybe it's the transcoding like you suggested, you'd be able to see a Plex Transcoding Process spike a core of your CPU, be sure to view by logical cores. However, you can monitor performance in Windows with Task Manager and Resource Monitor. It might show the type of playback, I don't remember. On Plex Media Server, you can see when another device plays. Then try again with much lowered bitrate/resolution. Try Direct Play, then Direct Steam, then Transcode. Second, your Plex-playback devices, force them into specific playback types. If it's not, at least you've isolated your networking for the time being. First, try and wire it even if you have to sling it across the house if that's the only issue, you can focus on your Wi-Fi. Seems like you're dealing with a few things at once.

Plex and hdhomerun