I've known for years that Culross, on the Firth of Forth in the Kingdom of Fife, is reputed to be the best-preserved example of a small Scottish town in the period from the late 16th to the 18th century. But until very recently, unlike millions of people worldwide, I'd never been there. That is to say that I've never watched a single episode of the hugely successful US TV series Outlander. Outlander, based on the novels of the American writer Diana Gabaldon, is set in Scotland, switching in time between the 18th century and the contemporary world, and frequently features scenes filmed in Culross.

Having finally spent a couple of days in Culross itself, my first report is that the royal burgh has successfully absorbed its new-found fame. Walking its narrow streets and lanes, one is reminded more than once of the Outlander connection, but the tone is measured and the actual impact seems very limited. No exploitation of the TV link – or indeed of earlier films featuring the town, such as Kidnapped and The 39 Steps – has occurred. There is nothing here of the Scottish kitsch which disfigures so many of Scotland's most popular tourist sites. Culross remains what it has always been: a quite remarkable reminder of an authentic Scottish past.

It's difficult to pin down what it is that makes Culross so special. It's a small place with a current population of around 400. Its houses and cottages are of no more than two storeys high. With their frequently white walls, they are attractively neat and unpretentious, and seem to display a natural, uniform elegance. Crow-stepped gables and deep red roof tiles are much in evidence. The streets and lanes – sometimes called wynds or vennels – are narrow and often roughly cobbled. Traffic is minimal. Visitors approaching from the Kincardine Bridge direction all park in a spacious riverside area before Culross itself is entered. This means that the prevailing atmosphere is astonishingly quiet and still. Apart from the odd car, occasionally a double-decker bus passes through on its way to Dunfermline. Taller than the houses on either side, its size makes it seem spectacularly out of place – like some elephant suddenly appearing in your High Street.

Crucially there seems to be no shops. I was accompanied on my two-day visit by Diane, my daughter-in-law, and India, her nine-year-old girl. On the first morning, we went out to pick up my unmissable Guardian – and found ourselves driving a few miles to the next village to find a shop selling it. No shops, but Culross is no kind of carefully-preserved but no longer functioning living space. It has a primary school, and clearly its homes are occupied by families who have been prepared to give up aspects of contemporary urban living in return for a more tranquil way of life. Still it is hard to exaggerate the difference the absence of rows of shops and shop windows seems to make. The sense of finding oneself stepping back into a different and quieter time is underlined.

Yet there is a telling irony here. In its heyday in the 17th and 18th centuries, Culross was in no sense at all a sleepy centre of calm tranquillity. Rather it was a thriving, thrusting and spectacularly modern royal burgh. In the late 16th century, Sir George Bruce, its leading merchant, had developed a coal mine that extended out under the water of the Firth of Forth – apparently the first of its kind in the world. Then it had a bustling trading and fishing harbour – of which nothing remains but a rickety slipway out into the firth. It was also a salt-panning and manufacturing centre, producing iron girdles for baking and cooking.

Today's Culross has no industry just as it has no shops. However, in the mid-18th century its almost 1,700 inhabitants supported 11 alehouses. By 1885, only two survived. Now its population – and its visitors – get by with just the Red Lion, an inn dating from the early 17th century, full of character and providing excellent food and drink. Two or three small cafes cater for the many visitors. Otherwise, what is enjoyable is just being there – wallking and looking and experiencing the atmosphere.

Of course, there are places to visit. Sir George Bruce's own mansion house, built and extended in the early 17th century, at some point acquired the title of Culross Palace, despite never having been any kind of royal residence. Its many rooms and garden have been beautifully restored by the National Trust for Scotland. The past comes vividly alive inside, while outside the garden with its mixture of fruit trees, flowers, bushes, vegetables, and free-ranging hens, is a real delight. In one corner, bags of potatoes and apples are on sale at £2.50. Please leave your payment in the box provided. (Subsequently we passed cottage doors offering passers-by homemade toffee or bottles of rose-hip syrup sitting in baskets attached to their doorknob. Again, just leave your payment in the basket. An older world indeed.)

The beautiful apartment we were living in was part of what had been a small chapel. No other church is in evidence. However, high above the town, and approached by a lengthy and steep lane, lies what is described as Culross Abbey. Built in the early Middle Ages, the original abbey did not survive as a result of changes in the style of its Cistercian order – and the impact of the Protestant reformation. Only a large range of impressive ruins are to be seen. On the other hand, the original Abbey Church eventually became a Church of Scotland parish church, and that remains its status now.

Alongside the Abbey is an ancient cemetery full of gravestones, the names and dates of which have long mouldered away. Some have masonic symbols and other similar indicators of the former occupations of their owners. Standing among the Abbey's ruins, and looking down across the town and out across the Firth of Forth, one senses here – as in so much of Culross – the reality of a different world and a different time.

Culross may have become a star of TV and film. The real thing is better still.