Householder orthogonalization plays an important role in numerical linear algebra. It attains perfect orthogonality regardless of the conditioning of the input. However, in the context of a non-standard inner product, it becomes difficult to apply Householder orthogonalization due to the lack of an initial orthogonal basis. We propose strategies to overcome this obstacle and discuss algorithms and variants of Householder orthogonalization with a non-standard inner product. Rounding error analysis and numerical experiments demonstrate that our approach is numerically stable.