I've noticed the same thing Charlie, namely, when that much weight is removed the performance goes up significantly. Yet we see Jeeps all the time that have 140 lb front bumpers, 120 lb winches, 180 lb rear bumper / tire carriers, and 37" to 40" spare tires that are mounted way up high and as far from the vehicles center of mass as possible. Talk about gauranteed screwed up handling and decreased performance, a 40" spare tire is certain to do it. Then add on the weight of rocker gaurds, steel flat fenders, skid plates, and all of your tools and gear in the back of the Jeep. Its a wonder the typical built up Jeep even moves at all. I found it was kind of hard to notice a difference when you added weight, but when you removed that extra weight again, that is when you noticed a night and day improvement in performance. In general, it must be easier to notice an improvement in performance, than it is to notice a decrease in performance.
As for "best performance upgrade", my money is on re-gearing the differentials. An auto trans JK with 35" tires and 3.73 gears, or with 37" tires and 4.10 gears, is going to be turning 1900 RPM at 75 mph on the highway. At 1900 RPM these engines only make 60 horsepower. By changing to 5.13 gears for the 35" tires, or 5.38 gears for the 37" tires, you increase engine RPM to 2600 at 75 mph. At 2600 RPM these engines make 90 horsepower. That is a 50% increase in power, for about $1000 spent on new gears.
2017 Jeep Wrangler Rubicon Recon; 35s - no lift, Warn 9.5 XP-S, ARB Twin Air, SPOD
2017 Jeep Grand Cherokee Trailhawk (wife's Jeep)
2016 Ram Laramie 3500 Megacab 4x4; 6.7 CTD, Aisen, AEV Prospector conversion; 37s, 4.10s, Warn 16.5K