Large Language Models (LLMs) have become backbone for conversational AI, code generation, summarization, and many more scenarios. However, their deployment poses significant challenges in environments where compute resources are limited mostly in hybrid cloud-fog architectures, where real-time inference may need to run closer to the edge.
In these instances, progressive model pruning plays a pivotal role offering solution to reduce model size and computation cost without impacting accuracy. In this article, we will discuss how to efficiently deploy LLMs across cloud-fog topologies using layer-aware, resource-adaptive pruning techniques.