/Engine/Source/Runtime/GeometryCore/Public/Spatial/SegmentTree3.h
The function compares potential hits (ie that pass the tolerance) with a sort of "distance" metric that tries to balance distance from the ray origin (ie "close to eye") and distance from the hit segment (ie "on the line"). There is no correct way to do this, currently the code uses a metric based on the opening angle between ray and segment points.
bool FindNearestVisibleSegmentHitByRay ( const FRay3d & Ray, [TFunctionRef](API\Runtime\Core\GenericPlatform\TFunctionRef)< bool(int32, const FVector3d &, const FVector3d &)> WithinToleranceCheck, [FSegment](API\Runtime\GeometryCore\Spatial\FSegmentTree3\FSegment) & NearestHitSegmentOut, [FRayNearestSegmentInfo](API\Runtime\GeometryCore\Spatial\FSegmentTree3\FRayNearestSegmentInfo) & NearestInfo, const [IMeshSpatial::FQueryOptions](API\Runtime\GeometryCore\Spatial\IMeshSpatial\FQueryOptions) & Options ) const