We consider the problem of Bayesian inference for parameters in non-linear regression models whereby the underlying unknown response functions are formed by a set of differential equations. Bayesian methods of inference for unknown parameters rely primarily on the posterior obtained by Bayes rule. For differential equation models, analytic and closed forms for the posterior are not available and one has to resort to approximations. We propose a two-stage Laplace expansion to approximate the marginal likelihood, and hence, the posterior, to obtain an approximate closed form solution. For large sample sizes, the method of inference borrows from non-linear regression theory for maximum likelihood estimates, and is therefore, consistent. Our approach is exact in the limit and does not need the specification of an additional penalty parameter. Examples in this paper include the exponential model and SIR (Susceptible-Infected-Recovered) disease spread model. © 2016 Author(s).