text: Title: [Train] Crash at end of training
  Body: ### What happened + What you expected to happen
  
  Recently I've been staring to experience a crash at the end of training. The backtrace is always the same:
  
  ```
  Training completed after 1 iterations at 2025-03-05 04:39:56. Total running time: 8min 51s
  2025-03-05 04:39:56,506	INFO tune.py:1009 -- Wrote the latest version of all result files and experiment state to 'earthdaily-pathfinders-scaleai/venus/afm-profiling/train/experimental-2025-03-05_04-31-02_a458' in 0.5035s.
  
  (TorchTrainer pid=439, ip=10.212.157.221) *** SIGSEGV received at time=1741149596 on cpu 63 ***
  (TorchTrainer pid=439, ip=10.212.157.221) PC: @     0x7f60b5a3c7be  (unknown)  ray::gcs::TaskInfoAccessor::AsyncAddTaskEventData()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b6ee7050       1824  (unknown)
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b591d975       1392  ray::core::worker::TaskEventBufferImpl::FlushEvents()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b58a66ec       1488  ray::core::CoreWorker::Disconnect()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b58a6a9d       1152  ray::core::CoreWorker::ForceExit()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b58a6ecf       1680  ray::core::CoreWorker::HandleKillActor()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b589e3d4        192  ray::rpc::ServerCallImpl<>::HandleRequestImpl()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b5c2bbc8       1168  EventTracker::RecordExecution()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b5c0fffe         48  std::_Function_handler<>::_M_invoke()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b5c10476        112  boost::asio::detail::completion_handler<>::do_complete()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b62d68db        128  boost::asio::detail::scheduler::do_run_one()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b62d8259        288  boost::asio::detail::scheduler::run()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b62d8962         96  boost::asio::io_context::run()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b57ff0b1       1280  ray::core::CoreWorker::RunIOService()
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b5d1d4e0         64  thread_proxy
  (TorchTrainer pid=439, ip=10.212.157.221)     @     0x7f60b6f341c4  (unknown)  (unknown)
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484: *** SIGSEGV received at time=1741149596 on cpu 63 ***
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484: PC: @     0x7f60b5a3c7be  (unknown)  ray::gcs::TaskInfoAccessor::AsyncAddTaskEventData()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b6ee7050       1824  (unknown)
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b591d975       1392  ray::core::worker::TaskEventBufferImpl::FlushEvents()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b58a66ec       1488  ray::core::CoreWorker::Disconnect()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b58a6a9d       1152  ray::core::CoreWorker::ForceExit()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b58a6ecf       1680  ray::core::CoreWorker::HandleKillActor()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b589e3d4        192  ray::rpc::ServerCallImpl<>::HandleRequestImpl()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b5c2bbc8       1168  EventTracker::RecordExecution()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b5c0fffe         48  std::_Function_handler<>::_M_invoke()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b5c10476        112  boost::asio::detail::completion_handler<>::do_complete()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b62d68db        128  boost::asio::detail::scheduler::do_run_one()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b62d8259        288  boost::asio::detail::scheduler::run()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b62d8962         96  boost::asio::io_context::run()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,568 E 439 479] logging.cc:484:     @     0x7f60b57ff0b1       1280  ray::core::CoreWorker::RunIOService()
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,569 E 439 479] logging.cc:484:     @     0x7f60b5d1d4e0         64  thread_proxy
  (TorchTrainer pid=439, ip=10.212.157.221) [2025-03-05 04:39:56,569 E 439 479] logging.cc:484:     @     0x7f60b6f341c4  (unknown)  (unknown)
  (TorchTrainer pid=439, ip=10.212.157.221) Fatal Python error: Segmentation fault
  (TorchTrainer pid=439, ip=10.212.157.221) 
  (TorchTrainer pid=439, ip=10.212.157.221) 
  (TorchTrainer pid=439, ip=10.212.157.221) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, charset_normalizer.md, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pyarrow.lib, pandas._libs.ops, pyarrow._compute, bottleneck.move, bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.tslibs.strptime, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._parquet, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, pyarrow._json, lazy_object_proxy.cext, matplotlib._c_internal_utils, PIL._imaging, matplotlib._path, kiwisolver._cext, matplotlib._image, _cffi_backend, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, sklearn.__check_build._check_build, sklearn.utils.murmurhash, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, sklearn.utils._openmp_helpers, sklearn.utils._logistic_sigmoid, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.utils._typedefs, sklearn.utils._readonly_array_wrapper, sklearn.metrics._dist_metrics, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.utils._cython_blas, sklearn.utils._heap, sklearn.utils._sorting, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction, sklearn.metrics._pairwise_fast, sklearn.utils._random, markupsafe._speedups, scipy.fftpack.convolve, tornado.speedups, greenlet._greenlet (total: 228)
  ```
  
  I am experiencing that regardless of the number of workers I use (one or multiple). I am always using the DDP strategy though. This is how I am initializing the PyTorch Lightning trainer in my training loop:
  
  ```
      trainer = Trainer(
          strategy=ray.train.lightning.RayDDPStrategy(),
          plugins=[ray.train.lightning.RayLightningEnvironment()],
  ```
  
  Beyond that, I'm not sure what information would be relevant, but I am happy to provide more info about the way I am running my training jobs upon request.
  
  The same backtrace has been reported before in a comment on [this issue](https://github.com/ray-project/ray/issues/49998), however the original description of that issue seems unrelated, so I am creating a new issue here.
  
  ### Versions / Dependencies
  
  Ray: 2.43.0
  
  I think I've only experienced this with Ray 2.43.0 and not in an older version.
  
  ### Reproduction script
  
  This is hard to reproduce - it happens occasionally at the end of training.
  
  ### Issue Severity
  
  Medium: It is a significant difficulty but I can work around it.