Okay, here is my first hand real life experience. I owned a 2007 JKUR that I did a 5.7 hemi conversion on. I ran it all over CO and Moab from 2008 thru 2012, give or take. I have not done the Rubicon yet, but I am pretty confident I did trails and obstacles as hard here in CO. Moab is its own unique beast, but did lots of hard trails there as well.
First off, the AEV hemi conversion kit is very well thought out and engineered. If the Jeep doesn't have the updated air intake box, add it. It really does lower IATs. The 545 trans is not great, but not horrible either.
You asked about trail experience. Don't let the internet repeaters tell you you MUST upgrade to D60s with a V8. That is nonsense. I ran the stock D44s with upgraded axle shafts and never had an issue. And while I used the skinny pedal appropriately, there are times when you have no choice but to use throttle input. Nothing ever broke. And that was with 37s, including MTR 37s which are true 37s, unlike KM2 37s which are really 35.5".
You didn't say what tire size you plan to run and or if you plan to retain the D30 up front. If going to 37s, I would probably upgrade to a take out D44 or the ProRock or some of the newer D44 aftermarket axles. Run decent shafts and you will be fine.
I personally loved my 5.7 on the trails. It never over heated. (Stories of hemi's with overheating issues usually involve long climbs up steep grades at interstate speeds at altitude. Mine never did over heat even then.) It climbed anything! After I sold it and took my 2013 JKUR to Moab with the exact same lift and tires, I struggled on difficult obstacles that I used to walk right up. Oh yeah, I don't have that extra weight in front anymore. Da!! It makes a difference. The extra mass from the hemi gave me increased front wheel traction. I sorely missed that in the 3.6 JKUR.
If you take good lines and use the gas pedal wisely I see no reason why the hemi would be a "money pit". And you will actually find climbing those granite slabs on the Rubicon to be even easier than with a stock motor JKU.
There is nothing inherently wrong with a well maintained 5.7 hemi JKU.