We investigate possible quantum ground states as well as the classical limit of a frustrated J(1)-J(2) Heisenberg model on the three-dimensional (3D) hyperhoneycomb lattice. Our study is inspired by the recent discovery of beta-Li2IrO3, where Ir4+ ions form a 3D network with each lattice site being connected to three nearest neighbors. We focus on the influence of magnetic frustration caused by the second-nearest-neighbor spin interactions. Such interactions are likely to be significant due to the large extent of 5d orbitals in iridates or other 5d transition metal oxides. In the classical limit, the ground state manifold is given by line degeneracies of the spiral magnetic-order wave vectors when J(2)/J(1) greater than or similar to 0.17 while the collinear stripy order is included in the degenerate manifold when J(2)/J(1) = 0.5. Quantum order-by-disorder effects are studied using both the semiclassical 1/S expansion in the spin-wave theory and the Schwinger-boson approach. In general, certain coplanar spiral orders are chosen from the classical degenerate manifold for a large fraction of the phase diagram. Nonetheless quantum fluctuations favor the collinear stripy order over the spiral orders in an extended parameter region around J(2)/J(1) = 0.5, despite the spin-rotation invariance of the underlying Hamiltonian. This is in contrast to the emergence of stripy order in the Heisenberg-Kitaev model studied earlier on the same lattice, where the Kitaev-type Ising interactions are important for stabilizing the stripy order. As quantum fluctuations become stronger, U(1) and Z(2) quantum spin liquid phases are shown to arise via quantum disordering of the Neel, stripy, and spiral magnetically ordered phases. The effects of magnetic anisotropy and their relevance to future experiments are also discussed.