In this sense, fine means thin. So a fine line is a very thin line, used figuratively to refer to a small difference or distinction between things:
There's a fine line between testing someone and entrapping them.
Sometimes there's only a very fine line between genius and madness.
A fine line exists between keeping it tasteful and going over the top.
=
The expression to walk a fine line is often used to mean "to achieve or maintain a balance" or "to be very close to two different ideas or attitudes at the same time":
He walked the fine line between confidence and cockiness.
Her family walks a fine line between modernity and fundamentalism.
Teachers walk a fine line of trust between their students and what their parents need to know.
