Search for ZZ and zγ* production in p(p)over-bar collisions at √s=1.96 TeV and limits on anomalous ZZZ and ZZγ* couplings

We present a study of mu mu mu mu, eeee, and mu mu ee events using 1 fb(-1) of data collected with the D0 detector at the Fermilab Tevatron p (p) over bar Collider at root s = 1.96 TeV. Requiring the lepton pair masses to be greater than 30 GeV, we observe one event, consistent with the expected background of 0.13 +/- 0.03 events and with the predicted standard model ZZ and Z gamma(*) production of 1.71 +/- 0.15 events. We set an upper limit on the ZZ and Z gamma(*) cross section of 4.4 pb at the 95% C.L. We also derive limits on anomalous neutral trilinear ZZZ and ZZ gamma(*) gauge couplings. The one-parameter 95% C.L. coupling limits with a form-factor scale Lambda = 1.2 TeV are -0.28 < f(40)(Z)< 0.28, -0.31 < f(50)(Z)< 0.29, -0.26 < f(40)(gamma)< 0.26, and -0.30 < f(50)(gamma)< 0.28.