Guru
•
8 Messages
Horrible packet loss/unwatchable - what do I do next?
OK, I'm a network guy but definitely more of a layer 3/WAN guy. I don't have extensive experience troubleshooting Ethernet. So I'm hoping some others here might point me in the right direction.
Sometime -- not sure exactly when, but probably within the next month -- my U-Verse live streams started stuttering/glitching badly. For a variety of reasons we just haven't been watching much TV recently, so I can't pinpoint when it started or what the circumstances were, but I can tell you how things are now.
Right now I am watching an HD stream on the main DVR. Nobody else in the house is watching anything. The stream looks fine, sounds fine, and U-Verse Realtime shows the stream is fine (maybe a couple of out-of-order packets, but we're talking 3 in a minute).
Now if I bring up the guide, scroll to another HD channel, and start something recording, my live stream is instantly unwatchable. Per U-Verse Realtime, EACH stream is now losing ~15% of packets.
Last night the problem was so bad I couldn't watch any live TV at all -- but I was able to watch a non-HD VOD show, oddly.
In all cases, U-Verse Realtime shows the WAN link as being golden -- I've had maybe 2 corrected blocks in the last hour, and they did not coincide with any of the video glitching anyway. So I'm guessing it's either an equipment problem or a local LAN problem. All set-top boxes are connected via Ethernet -- no coax/HPNA. I'm on the 32/5 WAN profile, and I've (theoretically) got 4HD/0SD, though I can't even test that.
I went through the router and cleaned up any/all port forwarding, in particular cutting down the timeouts -- I thought maybe the router was getting overloaded. I then rebooted the router -- that let me go from watching nothing to watching a single HD stream. But given that I couldn't even start a second HD stream right after reboot, I'm dubious that it's a router load issue.
So...any suggestions on where to go next? My core switch (an ASUS GigaX1024i) is a pseudo-managed switch -- it's not manageable using any standard tools but it has its own software. But it's limited in what I can do -- I can't even look at the MAC table. At this point if I can't think of anything else to try I'm going to start disabling LAN ports one at a time to see if a specific device (or devices -- there are small 4-port switches in a few rooms that cascade of the ASUS) is/are causing the problem. But I'd appreciate any insight anyone else might have.
Thanks in advance.
--chris
Accepted Solution
Official Solution
cheer
Guru
•
8 Messages
14 years ago
OK, problem solved. And it proved once again that I hatehatehate layer 1.
So my plan was to confirm or deny that it was my core switch at fault. But before I did that, I realized that in my office I had a pair of small 5-port switches, and since the RG plugged into once of them, and since that one plugged into the second one (which then plugged into a cable running to the core switch), I realized that either could be at fault. So I pulled both out -- directly connecting the RG to the core switch -- and the problem persisted.
From this I decided to assume that both little switches were fine. So I took one to the basement. I removed the cable that fed back to the RG and the cable that fed the DVR and plugged them into my little switch. I ran back upstairs...and the problem was gone. Fair enough, but just to make sure, I unplugged everything else from the big core switch and connected only the RG and DVR.
Problem was still gone. So now I was confident that the core switch was OK. I set the DVR to watch a live HD stream and started two other HD shows recording. I went back downstairs and, one by one, started connecting the other cables back to the core switch. I had my wife upstairs watch the TV and tell me when it started freezing up. I waited 60 seconds between each cable, just to be sure.
After about 8 or 9 cables, she called down that it was messed up. I remove the last cable I had connected and, after about 15 seconds, the TV cleared up again. I put that cable to the side and reconnected everything else. All was well.
Now, in theory all of my cables are tagged and I have a spreadsheet showing which goes where. In practice I started slacking off on that...plus, some of the tags had fallen off. All I could tell for sure was that it was one of the cables feeding the second floor bedrooms.
I ran upstairs and started checking the rooms to see if anything was down. My daughter's bedroom was OK -- she was able to watch something on her media PC that was sitting on the basement server. Ran into our bedroom and confirmed the U-Verse STB had connectivity. Ran into the guest bedroom, and nothing was even connected in there (so I wouldn't have gotten a link light). Finally my son's room...but he still had green flashing lights on the back of the STB.
This baffled me for a bit. I double-checked the other rooms and went back to my son's. I asked him if the U-Verse STB was the only thing connected to the Ethernet, and he said no -- his media PC was as well. He then pointed to the little switch. D'OH! That explained why the U-Verse STB still had link. I confirmed that the switch's connection to the wall was dead.
So now I knew it was either a device in his room or the wiring itself. I unplugged the ethernet cable from the wall, then ran back to the basement and reconnected the feed to the core switch. The TV immediately started freezing. Bingo -- a wiring problem.
I disconnected it again, then ran back upstairs. I pulled the wall plate off. Nothing was visibly wrong, so I snipped the cable, stripped it back, and wired up a new jack. I re-attached it to the wall, plugged my son's cable in, then went back to the basement and reconnected the feed to the core switch.
All is well. And that, folks, is why I HATE layer 1.
--chris
0
ScottMac
Former Employee
•
23 Messages
14 years ago
Try connecting one or two of the STBs directly to the RG and see if any of the issues change in character.
0
0
cheer
Guru
•
8 Messages
14 years ago
Ah, good idea. Can't believe I didn't think of that.
So I took the DVR into my office (where the RG is). Connected the DVR directly to the RG, and disconnected everything else (except the WAN cable, obviously).
And...it worked fine. Started two different HD recordings and watched a live stream -- no issues at all. Connected everything back up as before -- moving the DVR back into the family room -- and the problem came right back.
So it's definitely something on the network. Hurm.
0
0
SomeJoe7777
Expert
•
346 Messages
14 years ago
Could be your switch is going bad. I've had switches die on me, sometimes in inexplicable ways.
I have an SMC 48-port switch sitting on the floor beside me here at work I can't use. If all the ports are on one VLAN, it works fine. The instant you try to set a port to a different VLAN, the entire switch repeatedly crashes and reboots. No reason.
P.S. Glad to know someone used the Stream Analyzer for troubleshooting.
0
0
cheer
Guru
•
8 Messages
14 years ago
Yeah I'm leaning towards the switch at this point. In groups I went and shut off every other Ethernet device in the house, and the problem still occured. So it seems unlikely that it's a device on the network.
Unfortunately I don't have a spare switch to test with, so I'll likely have to go pick one up somewhere and try it.
And yeah, the Stream Analyzer was quite handy! Thanks for that.
0
0
oufanindallas
ACE - Master
•
480 Messages
14 years ago
Great persistant troubleshooting. Glad you got it fixed and you are up and running again.
*I am not a DIRECTV employee, and the views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
0
0
hpmsrm
Master
•
53 Messages
14 years ago
That is great news. I've been following your adventure and frustrations with interest. Persistence pays off. Please forgive me for chuckling a bit because your procedure sounds like the one I often find myself using......just plain old process-of-elimination. Good work.
.......Margaret Thatcher
0
0
docbombay
Guru
•
75 Messages
14 years ago
Since I am "cyberchallenged," I wouldn't have known where to begin (other than to post on the forum), let alone how to go about solving the problem. But I'm really impressed by your logical experimental approach--you make it sound so easy. I'm glad to hear you're up and running again, even if it did take a lot of busy work to get there!
docbombay
0
0
oufanindallas
ACE - Master
•
480 Messages
14 years ago
When I learned my electronics troubleshooting from the Navy 20+ years ago I learned about 1/2 splitting, check in the middle, do I have a gozinta and a gozouta, then it's in the back half of the piece of gear. As a technical support rep for a major software company now, I use the same principle when a customer calls. With our program it's either the data, the program or 3rd party applications. Once you eliminate 2 of 3 you know where to focus.
If that doesn't work, it's "Sir, do you have a copy of your Windows CD? "
*I am not a DIRECTV employee, and the views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
0
0