Hfrhyu

I'm trying to write an expression in which an accented expression enclosed by parentheses has an accent over the parentheses. For example,

vec(dotx)

This typesets fine when amsmath is not loaded, but with amsmath present, the x is typeset to the right of the dot, as if the inner expression is being treated as

dotx

This behavior only happens when a closing parentheses (or bracket or brace) is part of the argument to the outer accent.

The best workaround I've been able to find is to use overset for the outer accent instead of the actual accent command (or accentset).

Is this behavior expected/documented anywhere?

I believe that this question was previously asked at
amsmath possible bug: dot over bracketed dotx, but was incorrectly marked as a duplicate of dot over vec over : amsmath bug/feature?. The latter question was about the differences in typesetting between dotting a subscripted character and subscripting a dotted character.

This has to do with a long-standing amsmath issue, see Why do arguments to nested tilde or breve commands reappear when amsmath is used?

I can offer a workaround.

documentclassarticle
usepackageamsmath

makeatletter
newcommanddblacc[2]mathpalettedblacc@#1#2
newcommanddblacc@[2]dblacc@@#1#2
newcommanddblacc@@[3]%
 begingroup
 sboxz@$m@th#1#3$%
 #2boxz@%
 endgroup

makeatother

begindocument

begingather*
dblaccvec(dotx)
\
dblaccdotbeginbmatrix x \ dotx endbmatrix
\
dblaccvec(exp dotx)
endgather*

enddocument

A version that works also with accents:

documentclassarticle
usepackageamsmath
usepackageaccents

makeatletter
newsaveboxdblacc@boxa
newsaveboxdblacc@boxb
DeclareRobustCommanddblacc[2]mathpalettedblacc@#1#2
newcommanddblacc@[2]dblacc@@#1#2
newcommanddblacc@@[3]%
 begingroup
 sboxdblacc@boxa$m@th#1#3$%
 sboxdblacc@boxb$m@th#1#2copydblacc@boxa$%
 boxdblacc@boxb
 endgroup

makeatother

begindocument

begingather*
dblaccvec(dotx)quadscriptscriptstyledblaccvec(dotx)
\
dblaccdotbeginbmatrix x \ dotx endbmatrix
\
dblaccvec(exp dotx)
endgather*

enddocument

The following redefines an internal of amsmath. After that the output of your example looks good. I don't know whether this has any possibility of breaking other stuff. I've added every test case egreg used in his answer and the output looks good. It doesn't work with the accents package, though.

documentclass[]article

usepackage[]amsmath

makeatletter
protecteddefmathaccentV#1#2#3#4#5%
 %
 ifmmode
 mathaccentV@do#2#3#4#5%
 else
 @xpnonmatherr@csname #1endcsname
 fi
 
defmathaccentV@do#1#2#3#4%
 %
 globalletmacc@nucleus@empty
 mathaccent"accentclass@#1#2#3#4macc@nucleus
 
makeatother

begindocument
begingather*
vec(dotx)
\
dotbeginbmatrix x \ dotx endbmatrix
\
vec(exp dotx)
endgather*
enddocument