I just took mine off to replace the inside handle. There are 8 screws. At least the one behind the dash and the 4 on the bottom of the trim are going to be hard if not impossible to get to. All the trim pry bars in the world will not help there. Once the screws are out then you need to be able to release the trim tabs.
There are right angle screw drivers. You can try a small flat head bent at 90 degrees to loosen the screw. Get it loose enough you can stick your arm behind the door panel and get to the latch or rod.
There are no good door panels left to just go out and get another tan or late model xlt or Eddie Bauer wood trim specific door panel. They are an obsolete discontinued part and all the ones in the salvage yard are broken or ruined in some type of way.
I do not disagree but the screw that is a problem is right here. It is recessed about an inch as well. I tried to get the panel open enough to get the door handle off and it was not going to work. I can not see how to do it with the seat in the way as well.
I just noticed on rereading that he says the interior door handle is broken off. I was real close to getting mine replaced without pulling the panel. That might be the best bet. It will take effort but it is doable. I just do not see how you get the panel off with the door closed? Even enough to reach in looks like it would be hard.
You could always try a screw driver to open the inside handle, I presume it broke off near the mount. Just lever it open or if there is a good amount of plastic of the door handle left screw a anchor or deck screw into it then use pliers to open it as a handle.