On 3 April 2008, Burns was involved in a collision with a cyclist as he drove his 4x4 out of the Palace of Westminster gates into Parliament Square. The cyclist, British Army Major Stuart Lane, was thrown over the handlebars of his bicycle and broke two vertebrae in his neck "which could have hindered his Army career".

In February 2009 Burns pleaded guilty to a charge of careless driving and was fined £400 with £200 costs at the City of Westminster Magistrates' Court.

On 29 June 2010, Burns called Speaker of the House of Commons John Bercow, who stands at 5'6", a "stupid, sanctimonious dwarf" during a debate.

In November 2011, Burns compared members of the campaigning group 38 Degrees to zombies, which led to an open letter of complaint to Burns with over 85,000 signatures.

In January 2013, Burns, when the minister overseeing rail fares, was revealed as having been making use of a ministerial car instead of travelling the 35 miles from his home in Essex to London by train. He claimed this was because he was not allowed to read sensitive papers on the train, but his claim was immediately contradicted by the Cabinet Office, although it was conceded that reading Red Box matters in such an environment was not advisable. The cost to the taxpayer was estimated to be £80,000 per annum.