Old packages are not cleaned up

[ryanmoran]

What happened?

• What were you attempting to do?
  I am rebuilding my Python 3.8 app using Python 3.9. This worked fine, but when I exec into the container, I can see that my modules for Python 3.8 are still there alongside the modules that were installed for Python 3.9.

• What did you expect to happen?
  I expected that my app wouldn't contain old copies of modules that I'm not using.

• What was the actual behavior? Please provide log output, if possible.
  Here are the steps I followed, I'm using the integration/testdata/default_app in this case:

1. Build the app with Python 3.8

   pack build myapp --buildpack paketo-buildpacks/cpython --buildpack paketo-buildpacks/pip --buildpack paketo-buildpacks/pip-install --buildpack paketo-community/build-plan --env BP_CPYTHON_VERSION=3.8.*
   

2. Build the app with Python 3.9

   pack build myapp --buildpack paketo-buildpacks/cpython --buildpack paketo-buildpacks/pip --buildpack paketo-buildpacks/pip-install --buildpack paketo-community/build-plan --env BP_CPYTHON_VERSION=3.9.*
   

3. Exec into the container

   docker run -it --entrypoint launcher myapp bash
   

4. Confirm that the packages layer contains Python 3.8 and 3.9 modules

    ls -al /layers/paketo-buildpacks_pip-install/packages/lib/python3.*/site-packages
   

5. See output that looks like the following:

   /layers/paketo-buildpacks_pip-install/packages/lib/python3.8/site-packages:
   total 64
   drwxr-xr-x 16 cnb cnb 4096 Jan  1  1980 .
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 ..
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Flask-0.12.3.dist-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Jinja2-2.7.2-py3.8.egg-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 MarkupSafe-0.21-py3.8.egg-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Werkzeug-0.10.4.dist-info
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 click
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 click-8.0.1.dist-info
   drwxr-xr-x  5 cnb cnb 4096 Jan  1  1980 flask
   drwxr-xr-x  8 cnb cnb 4096 Jan  1  1980 gunicorn
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 gunicorn-19.5.0.dist-info
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 itsdangerous
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 itsdangerous-2.0.1.dist-info
   drwxr-xr-x  4 cnb cnb 4096 Jan  1  1980 jinja2
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 markupsafe
   drwxr-xr-x  5 cnb cnb 4096 Jan  1  1980 werkzeug
   
   /layers/paketo-buildpacks_pip-install/packages/lib/python3.9/site-packages:
   total 64
   drwxr-xr-x 16 cnb cnb 4096 Jan  1  1980 .
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 ..
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Flask-0.12.3.dist-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Jinja2-2.7.2-py3.9.egg-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 MarkupSafe-0.21-py3.9.egg-info
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 Werkzeug-0.10.4.dist-info
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 click
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 click-8.0.1.dist-info
   drwxr-xr-x  5 cnb cnb 4096 Jan  1  1980 flask
   drwxr-xr-x  8 cnb cnb 4096 Jan  1  1980 gunicorn
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 gunicorn-19.5.0.dist-info
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 itsdangerous
   drwxr-xr-x  2 cnb cnb 4096 Jan  1  1980 itsdangerous-2.0.1.dist-info
   drwxr-xr-x  4 cnb cnb 4096 Jan  1  1980 jinja2
   drwxr-xr-x  3 cnb cnb 4096 Jan  1  1980 markupsafe
   drwxr-xr-x  5 cnb cnb 4096 Jan  1  1980 werkzeug
   

Build Configuration

• What platform (pack, kpack, tekton buildpacks plugin, etc.) are you
  using? Please include a version.

  $ pack version
  0.20.0+git-66a4f32.build-2668
  

• What buildpacks are you using? Please include versions.

  paketo-buildpacks/cpython     0.7.2
  paketo-buildpacks/pip         0.5.1
  paketo-buildpacks/pip-install 0.1.3
  paketo-community/build-plan   0.0.12
  

• What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?

  base: Pulling from paketobuildpacks/builder
  Digest: sha256:6c42e7d119ef14742115fb31de8025bb981c7084e3ce4eed3444e3eef9a224cd
  

• Can you provide a sample app or relevant configuration (buildpack.yml,
  nginx.conf, etc.)?
  The sample app I used was this one: https://github.com/paketo-buildpacks/pip-install/tree/f0cad81e304b75d04fb0d0f89ae4926572f883db/integration/testdata/default_app.

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

[mterrel]

We recently ran into an issue that I believe is closely related. We're running on a little older versions of the buildpacks, but I think the basic issue still exists. For us, the issue showed up because one version of a build mistakenly installed the older asyncio package via requirements.txt. That package causes a Syntax Error when anyone imports asyncio on newer Python versions, so asyncio was then removed from requirements.txt. But because of the layer cache, asyncio remained installed in all future builds and so the app continued to Syntax Error. To my knowledge, there's no great way to remove an old, unused package from the cache other than removing the entire cache.

So my question/suggestion is if this issue might be solved in such a way as to treat the PIP layer cache more like a true cache, where the packages are available during the build process to speed things up, but only those packages that are actually needed are installed into the final built image?