Vince Alm is among those claiming a Great Britain team at the Games would undermine home nations' identities.
Welshman Bale has said he would like to represent Great Britain, with Fifa seeking to reassure individual nations.
But the Football Association of Wales (FAW) opposes its players featuring for Team GB at the London 2012 Olympics.
Great Britain will enter a men's football team for the first time since 1960 and will be coached by England Under-21 coach Stuart Pearce at the 2012 event.
However, Football Supporters Federation Cymru chairman Alm said there is widespread concern among fans.
"I really think he [Bale] has been ill-advised," Alm said of the Tottenham midfielder's support for the GB team.
"The majority of Welsh supporters are against it. You can get away with the England and Wales Cricket Board, you can get away with it with rugby, but football is totally different.
"This could be the end of the home unions and I think that would be sad.
"I can understand why Gareth has done it. We are Welsh and we are British. The main issue we have is with losing our international identity.
"The players need to be made aware of it if they've got aspirations of Team GB. They need to be told what the consequences are if there is a Team GB."
Wales face Norway in a friendly at the Cardiff City Stadium on 12 November and Bale is expected to be included in Gary Speed's squad.
Despite the controversy, Alm urged fans to support Bale, who has 32 caps for Wales.
"Let's not alienate ourselves from Gareth Bale," added Alm. "The way to deal with it [the controversy over the GB team] is through the FAW.
"Anyone who pulls on a Welsh shirt and goes out there and is playing football for his country deserves all the backing.
"He's a quality player and we need him at his best if we're going to continue to climb the rankings and have a chance of qualifying [for major tournaments]."