The Scripture just prior to the "keys" passage is near and dear to my heart. Here is Matt 16:13-18...

13 Now when Jesus came into the district of Caesarea Philippi, he asked his disciples, “Who do people say that the Son of Man is?” 14 And they said, “Some say John the Baptist, others say Elijah, and others Jeremiah or one of the prophets.” 15 He said to them, “But who do you say that I am?” 16 Simon Peter replied, “You are the Christ, the Son of the living God.” 17 And Jesus answered him, “Blessed are you, Simon Bar-Jonah! For flesh and blood has not revealed this to you, but my Father who is in heaven. 18 And I tell you, you are Peter, and on this rock I will build my church, and the gates of **** shall not prevail against it.

Listening to Peter's testimony of who Jesus actually is always warms my heart. Yet many church leaders turn this beautiful piece of Scripture into something it is not. They focus on Peter and not Jesus. They follow on with the "keys" passage to claim ultimate authority with the church, even beyond Christ as the Head of the Church. They fail on at least two key points:

1. The original Greek translates the middle of verse 18 as "and on this the rock I will build My church". Please notice the definite article "the" (τῇ) before the word "rock". Jesus' focus is not on Peter as many would have you believe, but rather on the Messiah Himself. "The rock" is Peter's answer to Jesus' question, "But who do you say that I am?" (emphasis mine) The authority that church fathers attempt to derive from this passage fails to manifest itself when Christ is the center of attention rather than Peter
2. Not only are the "keys" OF the kingdom and not TO the kingdom, as you rightfully point out, but the "binding" is clearly in response to God's will and not the will of man. Please take a look at this same Scripture in the Young's Literal Translation, "19 and I will give to thee the keys of the reign of the heavens, and whatever thou mayest bind upon the earth shall be having been bound in the heavens, and whatever thou mayest loose upon the earth shall be having been loosed in the heavens." English translations of the Bible often use active tense which tends to distort who the main actor is in a passage. The "binding" here is not the will of Peter and the other disciples, or of any other man, but rather of God Himself. The wording is admittedly a bit clunky, but hopefully you can see this principle in action if you pause mid-sentence with a comma... "whatever thou mayest bind upon the earth shall be, having been bound in the heavens". The binding and loosening has already taken place in Heaven before any "decision" was made by a disciple. I believe that it is God the Father who binds and loosens, and that the disciples were just following God's will