The average voter -- in particular, the average primary voter -- cares a lot about moral purity and expressive politics. So if you disempower the money, you empower the ideological purists who want candidates first and foremost to demonstrate fidelity to shared principles.
Everyone with a party affiliation (formal or informal) thinks this applies only to people of other parties. Bad news: it applies to you too.
Also, as I have opined in other venues, if you remove the money then you are left only with personal connections. Money is a much more transparent and quantifiable way to see who is influencing whom, than trying to decipher who is connected to whom.