Got error 500 when I click on the upgrade wizard in the admin menu. Renamed the upgrades/temp folder and now It re-work… so the temp folder was my issue…
cache/themes is recreated after you delete it. It is there for performance reasons. It is a feature
upload/upgrades stays there so you can retry upgrades and have a look at which files were changed. It’s part of normal maintenance to clean it up. It can be done by pressing “delete package” on the upgrades screen.
The only moments when they have to be manually deleted, or when that makes a difference, is after a botched upgrade that leaves an incoherent state behind. But that is unpredictable by definition…