This research presents LGTC-IPPO, a novel decentralized reinforcement learning approach designed for allocating diverse resources among multiple agents. The core innovation lies in its integration of dynamic cluster agreements, allowing agents to form temporary teams based on resource needs without relying on centralized control. Evaluated against other multi-agent reinforcement learning methods and even a centralized expert, the proposed system demonstrates enhanced performance, coordination, and stability in dynamic environments, particularly as the number of agents and resource types increases. The effectiveness of this decentralized strategy, especially in facilitating efficient resource reallocation when conditions change, highlights its potential for real-world applications where global coordination is challenging. The study also validates the approach through both extensive simulations and hardware experiments using drones.