I would worry. Go to a pool store and get a net that is made to remove debris - it works for ponds too. For future I would cover the pond with pond netting.

Besides filling up the pond the decaying leaves will cause massive algae bloom the following year(s). I would also suggest getting some of the benificial bacteria to put in your pond that will eat the decaying debris and help with this issue.