cache/themes is recreated after you delete it. It is there for performance reasons. It is a feature
upload/upgrades stays there so you can retry upgrades and have a look at which files were changed. It's part of normal maintenance to clean it up. It can be done by pressing "delete package" on the upgrades screen.
The only moments when they have to be manually deleted, or when that makes a difference, is after a botched upgrade that leaves an incoherent state behind. But that is unpredictable by definition...
My SuiteCRM In-depth blog. Thank you for always stating your SuiteCRM version, checking your logs, reading the Docs, and searching before you ask!
The administrator has disabled public write access.