Agree or disagree? Do you know the answer? Post a reply without even creating an account!
I'm installing pytorch to do some machine learning, and once again, I'm staggered by the enormous size of these packages. The wheel is over 800 megabytes, WHY?! And that's the compressed size!! I certainly understand that the models are large, and the data to train them even greater. But the software framework!? There can't possibly be that much actual code, it has to be something else, but what? and is it really needed?
And then there's nvidia cuda, which is an ever larger humongous beast weighing in at around three gigabytes for all the compressed wheels. How can this possibly make sense? It might include a lot of stuff optimized for all the different GPUs but come on! I really don't see why it needs to take up so much space. What's in it that is actually needed?
Have they just grown because both machine learning and game development (media files) deal with large amounts of data anyway so no one cares enough? Or is there an actual justification for the majority of the sizes?
According to a discussion specifically about pytorch being too large (and has grown since then) it seems that the majority of the size of pytorch is because it's compiled with nvidia cuda. The CPU-only version is significantly smaller. And the reason the included nvidia glue is large, and all the nvidia cuda packages are large, is because they include a version for every single nvidia GPU.
So it seems nvidia is the culprit. In my opinion this strategy seems wasteful and unsustainable. But they've been doing it for a long time now, despite continually getting larger and larger. However, there does appear to be a new compression-flag which reduces the size by 30-40%. It seems it's set to "speed" by default.