You can do something similar to an if/then/else in a regular expression, the syntax is (?if then|else). The else part is optional and could be removed with the vertical bar. The then and else part are subexpression of any kind. One of those will be evaluated depending on the if condition results. The if part is a condition and depnding on the implementation, it could be:

  • Back-references: In /(<B>)?This text may be bold(?(1)</B>)/, the (?(1)…) check that (<B>)? has match something, and if this is the case, try to match </B>.
  • Look-arounds: /(?(?<=a)b|c)/ means match b if precede by a else match c.
  • Host language code: Perl allow the use of arbitrary perl code has a conditional, for exemple, the following regurar expression matches correctly nested double angle quotes: 
   (?{local $nest=0})
          |  « (?{$nest++})
          |  » (?(?{$nest != 0}) (?{$nest--}) | (?!) ))
   (?(?{$nest!=0}) (?!))

For your conveninence, I have used the x mode for a better presentation. This is also the occasion to introduce the special syntax (?!) which litteraly says "fail now".
Finaly, using perl, there is another way to match nested contruct, which is called dynamic regex.