Highest resolution that the latest official firmware can support is 720x480 which is the same as your DVDs resolution. You just add aspect ratio signals to your encode so the PSP would know if it should be displayed in full widescreen or 4:3. This is useful if you're backing up your DVDs and want to use the same encode for both your PC and PSP. If you're only encoding for the PSP, then just use 480x272 resolution which is the PSP's screen resolution. Same with 720x480 encodes, you can add aspect-ratio signalling so the PSP knows whether to display it full widescreen or 4:3.
As for programs, there are a multitude of programs out there that you can use for conversion. My favorite is
3GP Converter. The default settings work ok, but fine tuning the settings to your liking is a somewhat tedious process since you have to edit "Transcoding.ini" manually via Notepad or similar. Other free programs you can use are
MeGUI,
PSPVC,
PSPVideo9, and
XviD4PSP. There are also for-pay programs available but I find that they're not configurable enough, and quality was not on par with the free programs I tested.