Answer by Phelype Oleinik for Adding optional argument (as exponent) to custom math expressions

You can use a flag value to represent “no value”. In this case, since it's for typesetting, it's pretty safe to use an empty default and test for that:

\newcommand\bigO[2][]%  {\mathcal{O}(#2\if\relax\detokenize{#1}\relax\else^{#1}\fi)}$\bigO{\epsilon}$ and $\bigO[3]{\epsilon}$

Or you can use \NewDocumentCommand, and test if the optional argument was given using \IfValueT:

\NewDocumentCommand \bigOo { o m }  {\mathcal{O}(#2\IfValueT{#1}{^{#1}})}$\bigOo{\epsilon}$ and $\bigOo[3]{\epsilon}$

Or if you want a more natural syntax, you can use the e argument type:

\NewDocumentCommand \bigOoh { m e^ }  {\mathcal{O}(#1\IfValueT{#2}{^{#2}})}$\bigOoh{\epsilon}$ and $\bigOoh{\epsilon}^3$

Here's a document with all three versions:

\documentclass{article}\begin{document}\newcommand\bigO[2][]%  {\mathcal{O}(#2\if\relax\detokenize{#1}\relax\else^{#1}\fi)}$\bigO{\epsilon}$ and $\bigO[3]{\epsilon}$\NewDocumentCommand \bigOo { o m }  {\mathcal{O}(#2\IfValueT{#1}{^{#1}})}$\bigOo{\epsilon}$ and $\bigOo[3]{\epsilon}$\NewDocumentCommand \bigOoh { m e^ }  {\mathcal{O}(#1\IfValueT{#2}{^{#2}})}$\bigOoh{\epsilon}$ and $\bigOoh{\epsilon}^3$\end{document}