Most people visit Toledo as a day trip from Madrid. Ideally, you should try and spend 2 full days and at least 2 nights in Toledo. One day is just not enough time for a city which was the capital of Spain before Madrid arrived on the scene. Toledo used to be the intersection of 3 major cultures of the world - Christianity, Islam, and Judaism - even today there are buildings and flavors in the streets of Toledo that will remind you of that amazing time in history.
Why do you need atleast 2 days?
The real Toledo awakens at night
Toledo is a hilly city
Most of Toledo’s charm lies in the steep streets and lanes up and down the hill on which the city is built. Going from one attraction to another can seem short on paper but when there is a climb involved, things take time. If you want to see most of the important sights and if you want to also get an experience of Toledo’s charm, you have to slowly navigate the hillside layout of the city. And this will need more time.
Enjoy the streets and Puertas
Too many things to see
Toledo is so much more than the Alcazar, Cathedral, and Synagogue. If you are even remotely interested in history or culture, churches or old markets, ancient Roman ruins or Moorish architecture, Toledo has enough to occupy even a week-long stay.
I would say that a Toledo experience of 2 days can be richer than a visit to Madrid for the same amount of time. So I encourage you to consider at least a 2 day stay in Toledo even if it means to sacrifice another day trip or a day from your stay in Madrid.