I'd agree with Wheat that the terms overlap significantly.
Avant garde - like many French words used in English - is a somewhat 'highbrow' term. In is traditional meaning, it is often used to describe 'high art' as well as political and philosophical movements that are seen to advance society in some way. Hence' to be avant garde in this sense, you need to be prominent enough in society to persuade society to break with the old ways (or at least, prominent enough in your field to persuade your colleagues to advance!)
To be experimental is also to put aside traditional techniques with known results, and to make use of a process in order to find out what the result of that process is. Unlike avant garde, being 'experimental' doesn't carry the connotation of moving society forward, or of your work having resonance with other spheres of human activity. In fact, 'experimental' seems to have been used as a dismissive term on occasion:
In the 1950s, the term "experimental" was often applied by
conservative music critics — along with a number of other words, such
as "engineers art", "musical splitting of the atom", "alchemist's
kitchen", "atonal", and "serial" — as a deprecating jargon term, which
must be regarded as "abortive concepts", since they did not "grasp a
subject". This was an attempt to marginalize, and thereby dismiss
various kinds of music that did not conform to established
conventions.
Used just on its own, avant garde tends to refer to classical-derived music, whereas experimental could refer to any music that is breaking new ground (and hence maybe does not currently have another genre label!) including some avant garde music (the 'dismissive' connotation didn't stick!)
Used in combination with other genre labels, avant garde loses some of its sense of pertaining to an advancement of society, so 'avant garde metal' and 'experimental metal' would largely mean the same thing. Nevertheless, if you're making music in your bedroom for yourself and two soundcloud listeners, it would seem quite pretentious to describe it as avant garde - better to stick to 'experimental' until you have an audience!