I had that problem too recently, luckily was able to fix it. Here's what I did:
1. Ensure you are using a wired Ethernet connection, with the cable going from your PC to the PS3. You
NEED to use wired connection settings on PS3 to make your life easier. Connect to the internet from your PC and create a shared network link between PC connection and PS3. There are loads of tuts on youtube how to do this.
2. Test internet connection settings and ensure a connection can be made. Ensure that the PS3 at least gets a IP address assigned to it when you test connection. Full internet access is not needed just to connect debugger and mess around with memory. You'll want full internet access to be able to connect to PSN / use mods online.
3. Go to the settings column on the XMB menu and go all the way to the bottom to "Debug Settings" and make sure all the following setting are selected. If you don't see these options and are on Rebug cfw, go into rebug toolbox and ensure the XMB type is set to '
EX'.
Release Check Mode > Development Mode
Boot Mode > System Software Mode
Network Settings for Debug > Single Setting
Restart your PS3 system if any changes were needed to be made.
4. Go into internet connection settings and check what IP address was assigned to your PS3. DEX cfw also have current IP in bottom right overlay of XMB menu. On PC, attempt to connect to that IP address from Target Manager using
DECR console settings. You will know when you successfully connect to your DEX PS3 when there will be a green circle icon next to your console name in Target Manager.
5. Start the Debugger. Assuming you correctly attached and have a debug eboot for your game you should be able to edit your memory from this point and attach rtm tools. If there is an error, continue reading.
6. If you get 'Please wait - target communications in progress...' window looping constantly without a connection, your PS3 is having an issue transferring some files to your PC. Use multiman / rebug toolbox / webman on PS3 to establish an FTP connection connect to PS3 IP address using FileZilla. Navigate to /dev_flash/sys/external and copy all of the files in that directory to a spot on your computer.
7. Close the debugger and target manager on your PC completely and navigate to your app data folder on your PC for the debugger
C:\Users\(Your Name)\AppData\Local\SN Systems\ProDG Debugger for PS3\flash_files
You should see a folder(s) in there with random numbers or letters.
8. Delete all folders in that directory, connect to PS3 again using Target manager and attempt to use the debugger again. You will see a new folder with random letters / numbers generate. Close out of target manager and debugger again and copy the files you got from the dev_flash directory from your PC earlier to the newly generated folder.
Debugger should work after this point!