The Prime Minister said it was “eccentric” to change governments at this stage but “I regret not to have been successful in those arguments”.
A new Tory leader will now be elected who will replace Mr Johnson in No 10.
“In politics, no one is remotely indispensable,” he acknowledged in a statement delivered from a lectern in Downing Street.
Watch shocking footage of drink and drug driver who caused death of young dad Richie Jordan as partner Carol King pleads for others not to make same mistake
Man arrested after collision leaves 13-year-old boy in critical condition
Sunderland rapist jailed for life as victims reveal horrific ordeal
Thunderstorms: When does the Met Office think thunder and lightning hit Sunderland this week and what does the yellow weather warning mean?
Sunderland AFC admits customer service has 'fallen well short of the required standard' and is 'completely unacceptable' as fans groups slam ticket office and club shop problems
“I want you to know how sad I am to be giving up the best job in the world, but them’s the breaks,” he said.
Mr Johnson intends to remain in No 10 until his successor is elected, but he faces resistance to that plan from within his own party and the Opposition.
He has already appointed new Cabinet ministers to replace MPs who quit as part of the mass ministerial exodus in protest at his leadership.
In a statement which was watched by staff, supportive MPs and his wife Carrie Johnson carrying their child Romy, the Prime Minister said: “It is clearly now the will of the parliamentary Conservative Party that there should be a new leader of that party and therefore a new prime minister.
“And I’ve agreed with Sir Graham Brady, the chairman of our backbench MPs, that the process of choosing that new leader should begin now and the timetable will be announced next week.
“And I’ve today appointed a Cabinet to serve, as I will, until a new leader is in place.”
But in a sign of the resentment he feels about being forced from office, less than three years after a landslide election win, Mr Johnson said: “In the last few days, I tried to persuade my colleagues that it would be eccentric to change governments when we’re delivering so much and when we have such a vast mandate and when we’re actually only a handful of points behind in the polls, even in mid-term after quite a few months of pretty relentless sledging and when the economic scene is so difficult domestically and internationally.
“I regret not to have been successful in those arguments and of course it’s painful not to be able to see through so many ideas and projects myself.
“But as we’ve seen, at Westminster the herd instinct is powerful, when the herd moves, it moves.”
The timetable for the leadership contest is expected to result in a successor being in place for the party’s conference in October – with Mr Johnson intending to stay in No 10 until the process is complete.
He said his successor’s priorities should include “cutting taxes because that is the way to generate the growth and the income we need to pay the great public services”.
“To that new leader, I say, whoever he or she may be, I say: ‘I will give you as much support as I can’.”
Tory leadership hopefuls had already begun setting out their stalls before Mr Johnson made his announcement.
Attorney General Suella Braverman and arch-Brexiteer Steve Baker have both indicated they will run.
Business Secretary Kwasi Kwarteng also hinted that he will throw his hat in the ring.
“We now need a new leader as soon as practicable. Someone who can rebuild trust, heal the country, and set out a new, sensible and consistent economic approach to help families,” he said.
Foreign Secretary Liz Truss cut short an official trip to Indonesia to return to Westminster and said: “We need calmness and unity now and to keep governing while a new leader is found.”
Tom Tugendhat, chairman of the Foreign Affairs Committee and a potential leadership candidate, said: “Now we need a clean start.”