Personal responsibility is very important, and I certainly wouldn't advocate for anyone believing that they can't achieve a lot if they put their mind to it. But there are two caveats to that. One is that no matter how successful you become in "the game" it does not free you from being the subject of discrimination. I know too many examples of this to name, including a personal friend who was appointed to be an Assistant Attorney General on his merits but was still told to get out of the neighborhood by the cops, despite the fact that he lived there. It was a ritzy place, ergo as a black guy, he didn't belong. Happens All. The. Time to the likes of Oprah and others.
The second, is that the legacy of poverty, violence, and despair can sap the will to live, much less the belief that one can actually get themselves free of all that by working hard. How well could you function if you'd only had one meal each day for the last week, hadn't gotten enough sleep because of noises on the street (or within your own home), etc., etc., etc. Blaming people for not doing better under conditions where'd you likely fair no better is unkind, and I'm not saying that you are doing that, but there is a larger inclination to do so. This is a part of what privilege means. If your family can't buy a home because they've been discriminated against in getting a loan (something that still widely happens even though it's illegal) for generations, then they can't build wealth and stability in the same way as those who can. And the cycle continues on a variety of levels, not just that one.
As Malcolm Gladwell described in Outliers, those who succeed have support, mentors, help, opportunities. No one actually pulls themselves up by their bootstraps particularly if they haven't even got any boots to begin with. Sure, people need to try to make the best of what they are given, but it doesn't erase pervasive societal dynamics that demand that some be lower in the pecking order so that some can be higher.