On the CI machines, the $HOME variable is redirected to the artifacts folder, which is under the repo root. The package cache is using $HOME. Thus between runs, the cache is removed.
To fix this, change the package cache on CI runs to be outside of the repo.