Search for Resonant and Nonresonant Higgs Boson Pair Production in the bb¯τ+τ- Decay Channel in pp Collisions at sqrt(s) = 13 TeV with the ATLAS Detector

A search for resonant and nonresonant pair production of Higgs bosons in the bb[over ¯]τ^+τ^- final state is presented. The search uses 36.1  fb^-1 of pp collision data with sqrt[s]=13  TeV recorded by the ATLAS experiment at the LHC in 2015 and 2016. Decays of the τ-lepton pairs with at least one τ lepton decaying to final states with hadrons and a neutrino are considered. No significant excess above the expected background is observed in the data. The cross-section times branching ratio for nonresonant Higgs boson pair production is constrained to be less than 30.9 fb, 12.7 times the standard model expectation, at 95% confidence level. The data are also analyzed to probe resonant Higgs boson pair production, constraining a model with an extended Higgs sector based on two doublets and a Randall-Sundrum bulk graviton model. Upper limits are placed on the resonant Higgs boson pair production cross-section times branching ratio, excluding resonances X in the mass range 305  GeV<402  GeV in the simplified hMSSM minimal supersymmetric model for tanβ=2 and excluding bulk Randall-Sundrum gravitons G_KK in the mass range 325  GeV<885  GeV for k/M[over ¯]_Pl=1.