Compare commits

..

60 Commits

Author SHA1 Message Date
dependabot[bot]
079c4cd232 build(deps): bump extern/googletest from 7140cd4 to 973323e
Bumps [extern/googletest](https://github.com/google/googletest) from `7140cd4` to `973323e`.
- [Release notes](https://github.com/google/googletest/releases)
- [Commits](7140cd416c...973323ed64)

---
updated-dependencies:
- dependency-name: extern/googletest
  dependency-version: 973323ed64a05b128418e7eab67016db5ba049df
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-01 14:53:00 +00:00
dependabot[bot]
8c60e8f1f7 build(deps): bump extern/googletest from 9156d4c to 7140cd4
Bumps [extern/googletest](https://github.com/google/googletest) from `9156d4c` to `7140cd4`.
- [Release notes](https://github.com/google/googletest/releases)
- [Commits](9156d4caac...7140cd416c)

---
updated-dependencies:
- dependency-name: extern/googletest
  dependency-version: a721f1b20c605f635413e22d8b7427a8b4f3956c
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-10 21:05:44 +02:00
Dennis Klein
a34fa64dc1 ci: pin third-party actions to release commit SHAs
- hendrikmuhs/ccache-action: the v1 major tag also lagged behind the
  latest release (v1.2.20 instead of v1.2.23), so this is a bump too
- threeal/cmake-action: v2 == v2.1.0, pinned as-is
2026-06-10 19:31:19 +02:00
Dennis Klein
ed87268394 ci: fix UBSan option spelling so it takes effect
- ENABLE_SANITIZER_UNDEFINED_BEHAVIOUR never matched the CMake option
  ENABLE_SANITIZER_UNDEFINED_BEHAVIOR (cmake/FairMQProjectSettings.cmake),
  so the asan+lsan+ubsan job has never actually enabled UBSan
2026-06-10 19:31:19 +02:00
Dennis Klein
7e63d4ae9a test: drop thread-sanitizer suppressions
- every entry stood in for a library tsan could not see into; with libzmq,
  libsodium and libstdc++ now tsan-instrumented in the tsan CI job, the
  happens-before edges they establish are visible and nothing is left to
  suppress
- suppressions were blunt (a race: entry matches any frame in the stack),
  so they could also mask real races passing through those frames
2026-06-10 19:31:19 +02:00
Dennis Klein
1fb5ad8c33 ci: switch thread-sanitizer job to gcc with instrumented deps
- build with the spack gcc toolchain like every other job: no clang++, no
  -fuse-ld=lld and no lld install step (the GNU BFD failure on tsan objects
  was specific to clang's tsan runtime)
- use gcc 15 for the freshest libtsan runtime; the asan entry stays on
  gcc 14, so the matrix now carries per-entry gcc and spack env names
- consume the tsan spack env and load the instrumented libstdc++ into each
  test's environment via FAIRMQ_TEST_LD_LIBRARY_PATH (it shares the soname
  of the compiler's own, so it substitutes process-wide at load time)
- use -fno-omit-frame-pointer for readable reports; optimization comes
  from the project's Debug -Og
- verify the wiring: assert the test environment resolves libstdc++ to the
  instrumented copy and that libzmq is tsan-instrumented, since both
  failure modes are silent (the suite still passes, with reduced race
  coverage)
2026-06-10 19:31:19 +02:00
Dennis Klein
add85cb18d ci: add tsan spack environment with instrumented libzmq
- mirror spack-latest.yaml, with -fsanitize=thread on the libzmq and
  libsodium nodes so tsan can observe the happens-before edges established
  inside libzmq's lock-free queues, plus the libstdcxx-tsan root spec
- flags are applied per node instead of via the propagating '==' operator,
  which could reach the gcc node and trigger a compiler rebuild
- unchanged roots (fairlogger, boost, ninja, cmake) keep their spec hashes,
  so they are shared with the regular buildcache entries; the instrumented
  nodes hash differently and coexist in the content-addressed cache
- exclude libstdcxx-tsan from concretizer reuse so recipe changes always
  take effect; unchanged recipes still hit the buildcache because the spec
  hash is identical
- add the tsan env to the buildcache matrix (rebuilding also on spack_repo
  changes) so the instrumented binaries are cached instead of rebuilt on
  every CI run
2026-06-10 19:31:19 +02:00
Dennis Klein
331c50ab0e ci: add spack package for a tsan-instrumented libstdc++
- gcc ships no supported switch to build libstdc++ with -fsanitize=thread,
  and spack's gcc recipe filters all flags out of the target-library build
  (CXXFLAGS_FOR_TARGET is owned by its generated --with-build-config=spack
  makefile), so provide a dedicated libstdcxx-tsan package in a custom repo
- build only the libstdc++-v3 subtree from the matching gcc release tarball,
  configured standalone against the already-installed toolchain (recipe
  modeled on https://iree.dev/developers/debugging/sanitizers/), instead of
  rebuilding all of gcc
- the result is a drop-in runtime replacement for the compiler's libstdc++
  (same soname and symbol versions), to be loaded only by the instrumented
  test executables
- normalize the install layout after make install: the standalone build puts
  the runtime libraries into the multilib os dir (lib64 on x86_64) regardless
  of --libdir, and --with-toolexeclibdir only applies to cross builds
- register the repo in the setup-deps action before creating the env
2026-06-10 19:31:19 +02:00
Dennis Klein
988ef81922 ci: tolerate cache-satisfied specs in the buildcache pushes
- buildcache push expands its selection into the full dependency
  closure, build-time dependencies included; specs that were satisfied
  from the buildcache do not have those installed locally, and the push
  fails with PackageNotInstalledError
- both push sites (the early gcc node push and the env-level push) only
  ever ran in fresh-build scenarios before, so the failure surfaced once
  the cache was warm
- pass --allow-missing to skip what is not installed (a best-effort push
  of everything that is); a freshly built gcc thus still uploads its
  build-time dependencies, which a future gcc rebuild can then pull as
  binaries
2026-06-10 19:31:19 +02:00
Dennis Klein
b568535910 test: support an alternative runtime library dir per test
- introduce FAIRMQ_TEST_LD_LIBRARY_PATH, which prepends a directory to
  each test's environment via ctest, so the tests can run against an
  alternative runtime library (e.g. a tsan-instrumented libstdc++)
- LD_LIBRARY_PATH rather than an injected rpath: an rpath added via the
  linker flags cannot precede the rpath spack's gcc adds through its
  specs file, so the compiler's own libstdc++ would keep winning the
  runtime search order
- scoped per test on purpose: an instrumented library has unresolved
  __tsan_* symbols and must not be loaded into uninstrumented tools
  like cmake, ctest or ninja
- fail the configuration instead of silently dropping the injection on
  CMake < 3.22 (ENVIRONMENT_MODIFICATION)
- cover the example tests too; they share the instrumented runtime but
  not the locale-cache warmup (their main() is the installed public
  header). The custom-controller env block was dead before: it tested
  lsan_options, which only ever existed in the add_example() function
  scope, so the test also never received the LSan suppressions
2026-06-10 19:31:19 +02:00
Dennis Klein
2febbe3146 build: append linker flags as strings, not lists
- CMAKE_EXE_LINKER_FLAGS and CMAKE_SHARED_LINKER_FLAGS are command-line
  strings; list(APPEND) inserts a semicolon once the variable is
  non-empty, which splits the link command at the shell level
- latent until now because nothing passed linker flags on the cmake
  command line
2026-06-10 19:31:19 +02:00
Dennis Klein
2bd9a072a9 test: pre-fill libstdc++ ctype caches before threads exist
- std::ctype<char> caches narrow()/widen() results per character in
  plain char arrays of the global classic-locale facet, written without
  synchronization from header-inlined code (locale_facets.h); two
  threads exercising an uncached character concurrently (e.g. compiling
  a std::regex in Channel::Validate) constitute a true data race that
  ThreadSanitizer rightfully reports
- the stores are real and unsynchronized, so a tsan-instrumented
  libstdc++ cannot help here; instead fill the caches before any thread
  is spawned, which turns every later access into a pure read
- warm the lazily-installed num_put/num_get caches used by stream
  insertion/extraction as well, via a small format/parse round-trip
- wire the warm-up into the gtest runner main() and, via a static
  initializer, into the test device runner
2026-06-10 19:31:19 +02:00
Dennis Klein
fc69b5e7ae refactor: compile channel name validation regex only once
- the pattern is constant; compiling it on every Validate() call is
  wasted work and, when channels are validated from multiple threads,
  needlessly exercises libstdc++'s lazily-populated ctype caches
2026-06-10 19:31:19 +02:00
Dennis Klein
19e607e486 test: fix racy loop-variable capture in SubscriptionThreadSafety
- the subscriber threads captured the loop counter by reference while
  the spawning loop kept incrementing it: a genuine data race
- depending on timing, threads could also end up with duplicate
  subscriber names; capture the counter by value instead
2026-06-10 19:31:19 +02:00
Dennis Klein
f08d42fcb8 fix: serialize console output in fair::mq::tools::execute
- concurrent execute() calls print captured subprocess lines to
  std::cout from multiple threads; the standard allows that, but
  libstdc++ maintains the formatted-output state (ios_base::width)
  with plain reads and writes -- a data race ThreadSanitizer reports
  once libstdc++ itself is instrumented
- a mutex around the insertion also keeps whole lines from
  interleaving
2026-06-10 19:31:19 +02:00
Alexey Rybalchenko
1597999aed fix(shmem): don't cache nullptr in GetRegionFromCache
A failed region lookup was inserted into the thread-local cache as
nullptr, making the failure permanent for the lifetime of the cache
generation - retrying never healed because the fast path would return
nullptr without calling GetRegion again. Skip the cache insert on
failure so subsequent calls retry the slow path.
2026-06-10 18:51:04 +02:00
Alexey Rybalchenko
215c31428b feat(shmem): expose side-channel metadata API for unsent messages
Add two public entry points needed by the ALICE use case where shmem
messages are allocated via a transport but never sent — their metadata
is instead serialised into Arrow tables and delivered over a separate
channel, allowing consumer devices to resolve the payload pointer
without taking ownership.

shmem::Message::GetMeta() returns the MetaHeader of the message,
mirroring the existing positional-init pattern already used in Socket.h.

shmem::GetDataAddressFromHandle(TransportFactory&, const MetaHeader&)
is a free function declared in Common.h and defined in Manager.cxx.
Keeping it out of the TransportFactory class body means callers only
need to include Common.h (available transitively via Message.h) and do
not drag in Socket.h or zmq.h. The implementation handles both managed
segments and unmanaged regions, and throws SharedMemoryError with a
typed message on a bad segment or region id. TransportFactory also
gains a same-named member for callers that already have the concrete
type. Lifetime of the returned pointer is the caller's responsibility;
the cache device is expected to hold the messages alive.

A SideChannel test covers the GetMeta/GetDataAddressFromHandle
round-trip for both standard and expanded-metadata configurations.
2026-06-10 18:51:04 +02:00
Dennis Klein
a0e8271aca test: suppress libzmq-induced thread-sanitizer false positives
- libzmq is not tsan-instrumented, so tsan cannot see the happens-before
  its queues establish between user threads and libzmq I/O threads,
  producing false-positive data races on message buffers
- add test/thread_sanitizer_suppressions.txt and point TSAN_OPTIONS at it
  via the sanitizers job env so it reaches the tests and their device
  subprocesses
- suppress: accesses made directly from libzmq, the zero-copy message
  deleters libzmq runs from msg_t::close, shmem receive-side metadata
  reads, and std::regex/locale lazy-init races in libstdc++
2026-06-09 23:00:58 +02:00
Dennis Klein
4b2c6cafac build: avoid -Wignored-attributes on the popen deleter
- glibc declares pclose with function attributes (nothrow/leaf), so
  decltype(pclose)* carries attributes that gcc ignores on the unique_ptr
  template argument, emitting -Wignored-attributes
- spell the deleter as a plain int(*)(FILE*) instead; pclose converts to
  it silently and the deleter behaves identically (both popen branches)
2026-06-09 23:00:58 +02:00
Dennis Klein
ab4bc49088 style: drop redundant member initializers in shmem UnmanagedRegion
- fShmemObject() and fFileMapping() default-construct anyway
- clang-tidy readability-redundant-member-init
  https://clang.llvm.org/extra/clang-tidy/checks/readability/redundant-member-init.html
2026-06-09 23:00:58 +02:00
Dennis Klein
89e16c0b19 build: run only the curated clang-tidy checks
- clang-tidy enables the clang-analyzer-* group by default; in this
  codebase it only yields false positives (intentional moved-from
  asserts, a bitmask enum cast) and noise inside third-party boost
  headers that HeaderFilterRegex does not filter
- prefix the check list with -* so only the explicitly curated checks run
2026-06-09 23:00:58 +02:00
Dennis Klein
35d713ebaa refactor: use default member initializers
- move constant constructor initializers into in-class default member
  initializers
- clang-tidy modernize-use-default-member-init
  https://clang.llvm.org/extra/clang-tidy/checks/modernize/use-default-member-init.html
2026-06-09 23:00:58 +02:00
Dennis Klein
4dfe24d411 style: drop redundant member initializers
- remove constructor member initializers that duplicate the default
  member initialization
- clang-tidy readability-redundant-member-init
  https://clang.llvm.org/extra/clang-tidy/checks/readability/redundant-member-init.html
2026-06-09 23:00:58 +02:00
Dennis Klein
d2aa2f10de style: drop redundant empty-string initializers
- remove `= ""` initialization from strings that default-construct empty
- clang-tidy readability-redundant-string-init
  https://clang.llvm.org/extra/clang-tidy/checks/readability/redundant-string-init.html
2026-06-09 23:00:58 +02:00
Dennis Klein
d5e0c29ced perf: emplace elements instead of inserting temporaries
- construct container elements in place instead of inserting a temporary
- clang-tidy modernize-use-emplace
  https://clang.llvm.org/extra/clang-tidy/checks/modernize/use-emplace.html
2026-06-09 23:00:58 +02:00
Dennis Klein
0feda158b4 perf: avoid copies in range-based for loops
- take loop variables by const reference where the body does not modify
  them
- clang-tidy performance-for-range-copy
  https://clang.llvm.org/extra/clang-tidy/checks/performance/for-range-copy.html
2026-06-09 23:00:58 +02:00
Dennis Klein
7a44c5e19e style: add braces around single-statement bodies
- wrap single-statement control-flow bodies in braces
- clang-tidy readability-braces-around-statements
  https://clang.llvm.org/extra/clang-tidy/checks/readability/braces-around-statements.html
2026-06-09 23:00:58 +02:00
Dennis Klein
e23c4d8ff1 style: use raw string literals for escaped strings
- replace string literals containing many escaped characters with raw
  string literals
- clang-tidy modernize-raw-string-literal
  https://clang.llvm.org/extra/clang-tidy/checks/modernize/raw-string-literal.html
2026-06-09 23:00:58 +02:00
Dennis Klein
8ab00ecddc refactor: use nullptr for null pointer literals
- replace `0` null pointer literals (e.g. `void* hint = 0`) with `nullptr`
- clang-tidy modernize-use-nullptr
  https://clang.llvm.org/extra/clang-tidy/checks/modernize/use-nullptr.html
2026-06-09 23:00:58 +02:00
Dennis Klein
b53d25738e build: curate clang-tidy to an enforceable check set
- drop the broad `cppcoreguidelines-*` glob: it produced ~4500 findings
  (magic numbers, non-private members, owning-memory, pointer arithmetic,
  ...) that are aspirational and out of scope for the warning gate
- drop modernize-use-equals-default: in this codebase it only yields
  false/unsafe positives, e.g. `= default` on a constructor that
  explicitly initializes atomic members (which default-init leaves
  indeterminate in C++17), and invalid output on constructors with a
  member-init list
- drop modernize-pass-by-value: it rewrites constructor parameters to
  by-value + std::move, changing public constructor signatures, which is
  an ABI-relevant change unsuitable for a library's public headers
- keep the deliberately-listed modernize/readability/performance checks
2026-06-09 23:00:58 +02:00
Dennis Klein
824825e911 ci: make the static-analysis warning gate actually fail
- the gate did `grep -q warning: build.log`, but build.log was never
  produced by the cmake-action build, so under `set -e` the grep in the
  `if` condition just reported "no match" and the job always passed
- as a result ~4961 clang-tidy warnings were silently ignored
- build manually and capture output to build.log with pipefail, and
  fail explicitly if the log is missing or contains a warning
2026-06-09 23:00:58 +02:00
Dennis Klein
2cf49cc50d ci: fix thread-sanitizer build with lld and PIC googletest
- tsan build failed at link with GNU ld:
  "failed to set dynamic section sizes: bad value" (known binutils +
  ThreadSanitizer incompatibility); install lld and select it via
  -fuse-ld=lld for the tsan job only
- pass -fuse-ld=lld through cxx-flags so it reaches the link line,
  avoiding the semicolon-list pitfall of
  list(APPEND CMAKE_EXE_LINKER_FLAGS ...)
- build the bundled googletest with CMAKE_POSITION_INDEPENDENT_CODE=ON:
  lld rejects R_X86_64_32 relocations from the non-PIC libgtest.a when
  producing the position-independent tsan executable; the bundle is built
  by a separate cmake invocation, so the flag must be set there
2026-06-09 23:00:58 +02:00
Dennis Klein
7e30c33bcf ci: raise locked-memory limit before running tests
- Region.PreallocateInsideSession.shmem and Example.region.shmem failed
  because mlock() of the managed segment/region hit RLIMIT_MEMLOCK on the
  runner ("Cannot allocate memory"); the region example then hung until
  its 30s timeout
- raise the limit via `sudo prlimit` in the same shell that launches
  ctest (per-process, so it must be done here, not in a prior step)
- replace threeal/ctest-action with the equivalent ctest invocation
2026-06-09 23:00:58 +02:00
Dennis Klein
0fd27cbbc3 ci: match renamed libzmq leak frame in lsan suppressions
- LSan symbolizes the leak as the C++ method `zmq::msg_t::init_size`
  in the Debug sanitizer build, no longer the C wrapper
  `zmq_msg_init_size`
- substring match failed (`_` vs `::`), so the suppression no longer
  applied and the asan+lsan+ubsan job failed in Pair/PubSub/Poller tests
- add the demangled frame, keep the old pattern for older libzmq
2026-06-09 23:00:58 +02:00
Dennis Klein
f374e228ff ci: cache gcc as a buildcache node instead of committed lockfiles
Committed lockfiles pinned gcc as a host-path external (from spack compiler
find), which is not portable across runners and broke CI. Cache the gcc
compiler itself as a buildcache node instead, so CI pulls it (~1 min) rather
than building it from source (~1 h).

- push the freshly-built gcc node in setup-deps BEFORE spack compiler find
  (which marks it external and excludes it from buildcache push), gated behind
  a push-gcc input used only by the buildcache workflow
- drop the committed-lockfile approach: remove test/ci/locks, the lockfile
  install path in setup-deps, and the lockfile export in the buildcache workflow
- drop the ignored ref input from setup-spack (v3 renamed it to spack_ref)
2026-06-08 23:04:29 +02:00
Dennis Klein
bb5c0a998c ci: install deps from committed lockfiles when present
Reusing concretization between the weekly buildcache (fresh) and weekday CI
(reuse) can drift if runner externals change, causing avoidable cache misses.

- setup-deps installs from test/ci/locks/<env>-gcc<N>.lock when it exists,
  skipping concretization for byte-identical hashes; falls back to the spec
  yaml otherwise
- buildcache exports each env's spack.lock as a downloadable artifact so the
  lockfiles can be regenerated on the ubuntu-24.04 runner and committed
- document the manual regeneration flow in test/ci/locks/README.md
2026-05-31 21:15:44 +02:00
Dennis Klein
ffc9c60f73 ci: cache fairmq compilation with ccache
FairMQ's own sources (library, examples, tests) were recompiled from scratch
in every matrix job on every push.

- add hendrikmuhs/ccache-action to build, sanitizers and static-analysis jobs
- set CMAKE_C/CXX_COMPILER_LAUNCHER=ccache so cmake routes through it
- key the cache per (job, env, gcc) since ccache hashes the compiler
2026-05-31 21:15:44 +02:00
Dennis Klein
1186bda040 ci: only run buildcache on dev/master pushes
The push trigger had a path filter but no branch filter, so any PR-branch push
touching those paths (e.g. a dependabot rebase pulling in setup-deps changes)
launched the full fresh buildcache matrix concurrently with CI.

- restrict the push trigger to branches [dev, master]
- frees runners for CI; the cache still refreshes via cron and on dev/master
2026-05-31 21:15:44 +02:00
Dennis Klein
14be1ce368 ci: fetch gcc from buildcache mirror and pin runner image
gcc was built from source (~58 min/job) because the FairMQ buildcache mirror
is only configured inside the env yaml, while gcc is installed before the env
is created.

- register the mirror globally after spack setup so "Install GCC" pulls the
  compiler as a binary
- pin runners to ubuntu-24.04 so the weekly buildcache and weekday CI share an
  image and concretize to matching hashes
- bump setup-spack to v3 to match the update-index job
2026-05-31 21:15:44 +02:00
dependabot[bot]
83cef5bdca build(deps): bump spack/setup-spack from 2 to 3
Bumps [spack/setup-spack](https://github.com/spack/setup-spack) from 2 to 3.
- [Release notes](https://github.com/spack/setup-spack/releases)
- [Changelog](https://github.com/spack/setup-spack/blob/main/CHANGELOG.md)
- [Commits](https://github.com/spack/setup-spack/compare/v2...v3)

---
updated-dependencies:
- dependency-name: spack/setup-spack
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-21 13:19:00 +02:00
dependabot[bot]
7f44fba4c0 build(deps): bump actions/checkout from 4 to 6
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-21 13:17:09 +02:00
dependabot[bot]
3dcfc29ec2 build(deps): bump extern/googletest from 7d76a23 to 04ee1b4
Bumps [extern/googletest](https://github.com/google/googletest) from `7d76a23` to `04ee1b4`.
- [Release notes](https://github.com/google/googletest/releases)
- [Commits](7d76a231b0...04ee1b4f2a)

---
updated-dependencies:
- dependency-name: extern/googletest
  dependency-version: 04ee1b4f2aefdffb0135d7cf2a2c519fe50dabe4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-05 14:47:59 +01:00
Dennis Klein
fa64faf3f7 fix(boost): add compatibility for Boost.Process v1 API in Boost 1.89+
Boost 1.88 replaced Boost.Process with v2, breaking the v1 API.
Boost 1.89 restores v1 compatibility via <boost/process/v1.hpp>.

- Fail configuration if Boost 1.88 is detected
- Define FAIRMQ_BOOST_PROCESS_V1_HEADER for Boost >= 1.89
- Use conditional includes to select v1.hpp or process.hpp
- Add namespace aliases (bp, bp_this) for portable API access
2026-01-05 14:11:19 +01:00
Dennis Klein
25abd605f3 ci: trigger buildcache on setup-deps changes 2025-12-01 09:29:24 +01:00
Dennis Klein
695ed89b6c ci: fix gcc version disambiguation in setup-deps
Use JSON output and jq to select the newest installed gcc version
when multiple versions match, avoiding conflicts between system and
spack-installed compilers.
2025-12-01 09:03:12 +01:00
Dennis Klein
cd074a3f1e ci: add boost187 variant to CI build matrix
Add boost187 environment variant to match the buildcache workflow,
using gcc-15 with the boost187 spack environment.
2025-12-01 08:58:20 +01:00
Dennis Klein
00c343858e ci: name setup-deps steps consistently 2025-11-30 21:03:27 +01:00
Dennis Klein
5dfeebba95 ci: add log grouping to setup-deps action 2025-11-30 20:59:46 +01:00
Dennis Klein
642a4e06f0 ci: force generic x86_64_v3 target for all packages 2025-11-30 20:48:13 +01:00
Dennis Klein
89f9f09c82 ci: deduplicate buildcache workflow using setup-deps action 2025-11-30 20:34:20 +01:00
Dennis Klein
a422361ee9 ci: set x86_64_v3 target for consistent buildcache 2025-11-30 20:10:50 +01:00
Dennis Klein
d1fbe4e89a ci: add named spack environments with boost187 variant 2025-11-30 19:58:19 +01:00
Dennis Klein
ef98c9c7ec ci: add --fresh flag to gcc installation
Force fresh concretization to avoid pulling gcc-runtime from public
buildcache which may be built with an incompatible gcc version.
2025-11-30 16:03:39 +01:00
Dennis Klein
399170879c ci: improve buildcache workflow
- rename mirror to ghcr-buildcache
- find system compiler before building gcc
- separate update-index job to avoid race condition
- always attempt push even on partial failure
2025-11-30 15:51:13 +01:00
Dennis Klein
1392a31250 ci: fix OCI registry authentication for buildcache push 2025-11-30 14:50:02 +01:00
Dennis Klein
060a5f2599 ci: add OCI registry login for buildcache push 2025-11-29 10:50:40 +01:00
Dennis Klein
7a8ccb8df6 ci: migrate from Jenkins to GitHub Actions 2025-11-28 21:13:30 +01:00
Dennis Klein
dcea48fcee fix: parse errors
```
/test/memory_resources/_memory_resources.cxx: In member function ‘virtual void {anonymous}::MemoryResources_allocator_Test::TestBody()’:
/test/memory_resources/_memory_resources.cxx:104:12: error: parse error in template argument list
  104 |     config.SetProperty<string>("session", to_string(session));
      |            ^~~~~~~~~~~~~~~~~~~
/test/memory_resources/_memory_resources.cxx:104:31: error: no matching function for call to ‘fair::mq::ProgOptions::SetProperty<<expression error> >(const char [8], std::string)’
  104 |     config.SetProperty<string>("session", to_string(session));
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/dklein/projects/FairMQ2/test/memory_resources/_memory_resources.cxx:11:
/fairmq/ProgOptions.h:269:6: note: candidate: ‘template<class T> void fair::mq::ProgOptions::SetProperty(const std::string&, T)’
  269 | void fair::mq::ProgOptions::SetProperty(const std::string& key, T val)
      |      ^~~~
/fairmq/ProgOptions.h:269:6: note:   template argument deduction/substitution failed:
/test/memory_resources/_memory_resources.cxx:104:31: error: template argument 1 is invalid
  104 |     config.SetProperty<string>("session", to_string(session));
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/test/memory_resources/_memory_resources.cxx: In member function ‘virtual void {anonymous}::MemoryResources_getMessage_Test::TestBody()’:
/test/memory_resources/_memory_resources.cxx:132:12: error: parse error in template argument list
  132 |     config.SetProperty<string>("session", to_string(session));
      |            ^~~~~~~~~~~~~~~~~~~
/test/memory_resources/_memory_resources.cxx:132:31: error: no matching function for call to ‘fair::mq::ProgOptions::SetProperty<<expression error> >(const char [8], std::string)’
  132 |     config.SetProperty<string>("session", to_string(session));
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/fairmq/ProgOptions.h:269:6: note: candidate: ‘template<class T> void fair::mq::ProgOptions::SetProperty(const std::string&, T)’
  269 | void fair::mq::ProgOptions::SetProperty(const std::string& key, T val)
      |      ^~~~
/fairmq/ProgOptions.h:269:6: note:   template argument deduction/substitution failed:
/test/memory_resources/_memory_resources.cxx:132:31: error: template argument 1 is invalid
  132 |     config.SetProperty<string>("session", to_string(session));
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
2025-06-13 08:17:53 +02:00
Giulio Eulisse
67dcf77a7f De-boostify: use std::pmr from C++17 2025-06-13 08:02:06 +02:00
Alexey Rybalchenko
24e7a5b8d0 Make shmem headers public 2025-03-17 15:16:46 +01:00
64 changed files with 1003 additions and 400 deletions

View File

@@ -1,3 +1,3 @@
---
Checks: 'cppcoreguidelines-*,misc-unused-alias-decls,misc-unused-parameters,modernize-pass-by-value,modernize-deprecated-headers,modernize-raw-string-literal,modernize-redundant-void-arg,modernize-use-bool-literals,modernize-use-default-member-init,modernize-use-emplace,modernize-use-equals-default,modernize-use-equals-delete,modernize-use-noexcept,modernize-use-nullptr,modernize-use-override,modernize-use-using,performance-faster-string-find,performance-for-range-copy,performance-unnecessary-copy-initialization,readability-avoid-const-params-in-decls,readability-braces-around-statements,readability-container-size-empty,readability-delete-null-pointer,readability-redundant-member-init,readability-redundant-string-init,readability-static-accessed-through-instance,readability-string-compare'
Checks: '-*,misc-unused-alias-decls,misc-unused-parameters,modernize-deprecated-headers,modernize-raw-string-literal,modernize-redundant-void-arg,modernize-use-bool-literals,modernize-use-default-member-init,modernize-use-emplace,modernize-use-equals-delete,modernize-use-noexcept,modernize-use-nullptr,modernize-use-override,modernize-use-using,performance-faster-string-find,performance-for-range-copy,performance-unnecessary-copy-initialization,readability-avoid-const-params-in-decls,readability-braces-around-statements,readability-container-size-empty,readability-delete-null-pointer,readability-redundant-member-init,readability-redundant-string-init,readability-static-accessed-through-instance,readability-string-compare'
HeaderFilterRegex: '/(fairmq/)'

78
.github/actions/setup-deps/action.yml vendored Normal file
View File

@@ -0,0 +1,78 @@
name: Install dependencies
description: Setup spack and install dependencies
inputs:
gcc:
description: 'GCC version to use'
required: true
env:
description: 'Spack environment name (latest, boost187, tsan)'
default: 'latest'
fresh:
description: 'Use fresh concretization'
default: 'false'
push-gcc:
description: 'Push the freshly-built gcc node to the buildcache (buildcache workflow only)'
default: 'false'
push-token:
description: 'Token with packages:write, required when push-gcc is true'
default: ''
runs:
using: composite
# Composite action step names are not shown in the UI (https://github.com/actions/runner/issues/1877),
# so we use ::group:: to make them visible.
steps:
- name: Setup spack
uses: spack/setup-spack@v3
with:
color: true
buildcache: true
- name: Find system compiler
shell: spack-bash {0}
run: |
echo "::group::Find system compiler"
spack compiler find
echo "::endgroup::"
- name: Add FairMQ buildcache mirror
shell: spack-bash {0}
run: |
echo "::group::Add FairMQ buildcache mirror"
spack mirror add --unsigned --type binary \
ghcr-buildcache oci://ghcr.io/fairrootgroup/fairmq-spack-buildcache
echo "::endgroup::"
- name: Install GCC
shell: spack-bash {0}
env:
GITHUB_TOKEN: ${{ inputs.push-token }}
run: |
echo "::group::Install GCC"
spack install ${{ inputs.fresh == 'true' && '--fresh' || '' }} gcc@${{ inputs.gcc }} target=x86_64_v3
gcc_hash=$(spack find --json gcc@${{ inputs.gcc }} target=x86_64_v3 | jq -r 'sort_by(.version | split(".") | map(tonumber)) | last | .hash')
if [ "${{ inputs.push-gcc }}" = "true" ]; then
# Push the gcc node now, BEFORE `spack compiler find` registers it as an
# external -- externals are excluded from `buildcache push`. This is what
# lets CI pull gcc from the cache (~1 min) instead of building it (~1 h).
# --allow-missing: when gcc itself was pulled from the cache, its
# build-time dependencies are not installed locally and would
# otherwise fail the push.
spack mirror set --oci-username ${{ github.actor }} --oci-password-variable GITHUB_TOKEN ghcr-buildcache
spack buildcache push --unsigned --allow-missing ghcr-buildcache /$gcc_hash
fi
spack compiler find "$(spack location -i /$gcc_hash)"
echo "::endgroup::"
- name: Install dependencies
shell: spack-bash {0}
run: |
echo "::group::Install dependencies"
spack repo add "$GITHUB_WORKSPACE/test/ci/spack_repo/fairmq_ci"
spack env create fairmq test/ci/spack-${{ inputs.env }}.yaml
spack -e fairmq add gcc@${{ inputs.gcc }}
spack -e fairmq config add "packages:all:require:'%gcc@${{ inputs.gcc }}'"
spack -e fairmq install --fail-fast ${{ inputs.fresh == 'true' && '--fresh' || '' }}
spack env activate --sh fairmq | grep '^export ' | sed 's/^export //;s/;$//' >> $GITHUB_ENV
echo "::endgroup::"

82
.github/workflows/buildcache.yml vendored Normal file
View File

@@ -0,0 +1,82 @@
name: Spack Buildcache
on:
workflow_dispatch:
schedule:
- cron: '0 3 * * 0' # Weekly on Sunday at 3am UTC
push:
branches: [dev, master]
paths:
- 'test/ci/spack-*.yaml'
- 'test/ci/spack_repo/**'
- '.github/workflows/buildcache.yml'
- '.github/actions/setup-deps/**'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
if: github.repository == 'FairRootGroup/FairMQ'
name: ${{ matrix.env }}-gcc-${{ matrix.gcc }}
runs-on: ubuntu-24.04
permissions:
packages: write
strategy:
fail-fast: false
matrix:
gcc: ['12', '13', '14', '15']
env: ['latest']
include:
- gcc: '15'
env: 'boost187'
- gcc: '15'
env: 'tsan'
steps:
- uses: actions/checkout@v6
- name: Setup spack environment
uses: ./.github/actions/setup-deps
with:
gcc: ${{ matrix.gcc }}
env: ${{ matrix.env }}
fresh: 'true'
push-gcc: 'true'
push-token: ${{ secrets.GITHUB_TOKEN }}
- name: Push to buildcache
if: ${{ !cancelled() }}
shell: spack-bash {0}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
spack -e fairmq mirror set --oci-username ${{ github.actor }} --oci-password-variable GITHUB_TOKEN ghcr-buildcache
# --allow-missing: build-time dependencies of specs that were
# satisfied from the buildcache are not installed locally and
# would otherwise fail the whole push.
spack -e fairmq buildcache push --unsigned --allow-missing ghcr-buildcache
update-index:
if: github.repository == 'FairRootGroup/FairMQ' && !cancelled()
needs: build
runs-on: ubuntu-24.04
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
permissions:
packages: write
steps:
- uses: actions/checkout@v6
- name: Setup spack
uses: spack/setup-spack@v3
with:
color: true
- name: Update buildcache index
shell: spack-bash {0}
run: |
spack env create fairmq test/ci/spack-latest.yaml
spack -e fairmq mirror set --oci-username ${{ github.actor }} --oci-password-variable GITHUB_TOKEN ghcr-buildcache
spack -e fairmq buildcache update-index ghcr-buildcache

View File

@@ -22,7 +22,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Try updating metadata
run: python meta_update.py
- name: Check for Updates

209
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,209 @@
name: CI
on:
push:
branches: [master, dev]
pull_request:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
if: github.repository == 'FairRootGroup/FairMQ'
name: ${{ matrix.env }}-gcc-${{ matrix.gcc }}
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
gcc: ['12', '13', '14', '15']
env: ['latest']
include:
- gcc: '15'
env: 'boost187'
steps:
- uses: actions/checkout@v6
with:
submodules: true
fetch-tags: true
fetch-depth: 0
- name: Setup spack environment
uses: ./.github/actions/setup-deps
with:
gcc: ${{ matrix.gcc }}
env: ${{ matrix.env }}
- name: ccache
uses: hendrikmuhs/ccache-action@d62db5f07c26379fc4b4e0916f098a92573c3b03 # v1.2.23
with:
key: ${{ github.job }}-${{ matrix.env }}-gcc${{ matrix.gcc }}
max-size: 500M
- name: Configure and Build
uses: threeal/cmake-action@725d1314ccf9ea922805d7e3f9d9bcbca892b406 # v2.1.0
with:
generator: Ninja
options: |
CMAKE_BUILD_TYPE=RelWithDebInfo
CMAKE_INSTALL_PREFIX=${{ github.workspace }}/install
BUILD_TESTING=ON
CMAKE_C_COMPILER_LAUNCHER=ccache
CMAKE_CXX_COMPILER_LAUNCHER=ccache
- name: Test
run: |
# Region/segment tests mlock() shared memory; raise the locked-memory
# limit so it does not hit RLIMIT_MEMLOCK on the runner.
sudo prlimit --pid $$ --memlock=unlimited:unlimited
ulimit -l
ctest --test-dir build --output-on-failure --no-tests=error
- name: Install
run: cmake --install build
sanitizers:
if: github.repository == 'FairRootGroup/FairMQ'
name: ${{ matrix.sanitizer.name }}
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
sanitizer:
- name: asan+lsan+ubsan
gcc: '14'
env: latest
options: |
ENABLE_SANITIZER_ADDRESS=ON
ENABLE_SANITIZER_LEAK=ON
ENABLE_SANITIZER_UNDEFINED_BEHAVIOR=ON
cxx-flags: -O1 -fno-omit-frame-pointer
- name: tsan
gcc: '15'
env: tsan
options: ENABLE_SANITIZER_THREAD=ON
cxx-flags: -fno-omit-frame-pointer
steps:
- uses: actions/checkout@v6
with:
submodules: true
fetch-tags: true
fetch-depth: 0
- name: Setup spack environment
uses: ./.github/actions/setup-deps
with:
gcc: ${{ matrix.sanitizer.gcc }}
env: ${{ matrix.sanitizer.env }}
- name: ccache
uses: hendrikmuhs/ccache-action@d62db5f07c26379fc4b4e0916f098a92573c3b03 # v1.2.23
with:
key: ${{ github.job }}-${{ matrix.sanitizer.name }}
max-size: 500M
- name: Locate instrumented libstdc++
if: matrix.sanitizer.name == 'tsan'
shell: spack-bash {0}
# The test processes must load the tsan-instrumented libstdc++
# instead of the compiler's own (same soname, LD_LIBRARY_PATH beats
# the RUNPATH). Set per test via ctest, not at the job level: like
# any shared library built with gcc -fsanitize=thread it has
# unresolved __tsan_* symbols, so loading it into uninstrumented
# tools (cmake, ctest, ninja) would break them.
run: |
prefix=$(spack -e fairmq location -i libstdcxx-tsan)
echo "test_library_path=$prefix/lib" >> $GITHUB_ENV
- name: Configure and Build
uses: threeal/cmake-action@725d1314ccf9ea922805d7e3f9d9bcbca892b406 # v2.1.0
with:
generator: Ninja
cxx-flags: ${{ matrix.sanitizer.cxx-flags }}
options: |
CMAKE_BUILD_TYPE=Debug
BUILD_TESTING=ON
CMAKE_C_COMPILER_LAUNCHER=ccache
CMAKE_CXX_COMPILER_LAUNCHER=ccache
FAIRMQ_TEST_LD_LIBRARY_PATH=${{ env.test_library_path }}
${{ matrix.sanitizer.options }}
- name: Verify tsan instrumentation wiring
if: matrix.sanitizer.name == 'tsan'
shell: spack-bash {0}
run: |
set -x
# the test environment must resolve libstdc++ to the instrumented copy
LD_LIBRARY_PATH=$test_library_path ldd build/test/testsuite_Channel \
| grep 'libstdc++' | tee /dev/stderr | grep -q libstdcxx-tsan
# libzmq must be instrumented
nm -D --undefined-only "$(spack -e fairmq location -i libzmq)/lib/libzmq.so" \
| grep -q __tsan_
# the instrumented libstdc++ must match the compiler release exactly,
# or binaries may reference GLIBCXX versions the runtime lacks
# (--color=never: the CI config forces SPACK_COLOR=always, which
# would wrap the version in ANSI escapes)
test "$(spack --color=never -e fairmq find --format '{version}' libstdcxx-tsan)" \
= "$(g++ -dumpfullversion)"
# the LD_LIBRARY_PATH prepend must reach the registered tests
# (via a file: grep -q quits on first match, and SIGPIPE on the
# large json output would fail the step under pipefail)
ctest --test-dir build --show-only=json-v1 > ctest-show-only.json
grep -q libstdcxx-tsan ctest-show-only.json
- name: Test
run: |
# Region/segment tests mlock() shared memory; raise the locked-memory
# limit so it does not hit RLIMIT_MEMLOCK on the runner.
sudo prlimit --pid $$ --memlock=unlimited:unlimited
ulimit -l
ctest --test-dir build --output-on-failure --no-tests=error
static-analysis:
if: github.repository == 'FairRootGroup/FairMQ'
name: static-analysis
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
submodules: true
fetch-tags: true
fetch-depth: 0
- name: Setup spack environment
uses: ./.github/actions/setup-deps
with:
gcc: '14'
- name: ccache
uses: hendrikmuhs/ccache-action@d62db5f07c26379fc4b4e0916f098a92573c3b03 # v1.2.23
with:
key: ${{ github.job }}
max-size: 500M
- name: Configure
run: |
cmake -S . -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DBUILD_TESTING=ON \
-DRUN_STATIC_ANALYSIS=ON \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
- name: Build
run: |
set -o pipefail
cmake --build build 2>&1 | tee build.log
- name: Check for warnings
run: |
test -f build.log || { echo "::error::build.log was not produced"; exit 1; }
if grep -q "warning:" build.log; then
echo "::error::Static analysis found warnings"
grep "warning:" build.log
exit 1
fi

View File

@@ -16,6 +16,6 @@ jobs:
container:
image: gitlab-registry.in2p3.fr/escape2020/wp3/eossr:v1.0
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: validate codemeta
run: eossr-metadata-validator codemeta.json

View File

@@ -46,6 +46,13 @@ include(FairMQDependencies)
# Targets ######################################################################
if(FAIRMQ_TEST_LD_LIBRARY_PATH AND CMAKE_VERSION VERSION_LESS 3.22)
# The per-test injection relies on ctest's ENVIRONMENT_MODIFICATION, which
# older CMake silently drops -- the tests would run against the default
# runtime while looking green.
message(FATAL_ERROR "FAIRMQ_TEST_LD_LIBRARY_PATH requires CMake >= 3.22")
endif()
if(BUILD_FAIRMQ)
add_subdirectory(fairmq)
endif()

View File

@@ -1,93 +0,0 @@
################################################################################
# Copyright (C) 2021-2023 GSI Helmholtzzentrum fuer Schwerionenforschung GmbH #
# #
# This software is distributed under the terms of the #
# GNU Lesser General Public Licence (LGPL) version 3, #
# copied verbatim in the file "LICENSE" #
################################################################################
cmake_host_system_information(RESULT fqdn QUERY FQDN)
set(CTEST_SOURCE_DIRECTORY .)
set(CTEST_BINARY_DIRECTORY build)
set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_USE_LAUNCHERS ON)
set(CTEST_CONFIGURATION_TYPE "RelWithDebInfo")
set(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE 102400)
if(NOT NCPUS)
if(ENV{SLURM_CPUS_PER_TASK})
set(NCPUS $ENV{SLURM_CPUS_PER_TASK})
else()
include(ProcessorCount)
ProcessorCount(NCPUS)
if(NCPUS EQUAL 0)
set(NCPUS 1)
endif()
endif()
endif()
if ("$ENV{CTEST_SITE}" STREQUAL "")
set(CTEST_SITE "${fqdn}")
else()
set(CTEST_SITE $ENV{CTEST_SITE})
endif()
if ("$ENV{LABEL}" STREQUAL "")
set(CTEST_BUILD_NAME "build")
else()
set(CTEST_BUILD_NAME $ENV{LABEL})
endif()
ctest_start(Continuous)
list(APPEND options "-DDISABLE_COLOR=ON" "-DBUILD_EXAMPLES=ON" "-DBUILD_TESTING=ON")
if(RUN_STATIC_ANALYSIS)
list(APPEND options "-DRUN_STATIC_ANALYSIS=ON")
endif()
if(CMAKE_BUILD_TYPE)
set(CTEST_CONFIGURATION_TYPE ${CMAKE_BUILD_TYPE})
endif()
if(ENABLE_SANITIZER_ADDRESS)
list(APPEND options "-DENABLE_SANITIZER_ADDRESS=ON")
endif()
if(ENABLE_SANITIZER_LEAK)
list(APPEND options "-DENABLE_SANITIZER_LEAK=ON")
endif()
if(ENABLE_SANITIZER_UNDEFINED_BEHAVIOR)
list(APPEND options "-DENABLE_SANITIZER_UNDEFINED_BEHAVIOR=ON")
endif()
if(ENABLE_SANITIZER_MEMORY)
list(APPEND options "-DENABLE_SANITIZER_MEMORY=ON")
endif()
if(ENABLE_SANITIZER_THREAD)
list(APPEND options "-DENABLE_SANITIZER_THREAD=ON")
endif()
if(CMAKE_CXX_COMPILER)
list(APPEND options "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}")
endif()
if(CMAKE_CXX_FLAGS)
list(APPEND options "-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}")
endif()
list(REMOVE_DUPLICATES options)
list(JOIN options ";" optionsstr)
ctest_configure(OPTIONS "${optionsstr}")
ctest_submit()
ctest_build(FLAGS "-j${NCPUS}")
ctest_submit()
if(NOT RUN_STATIC_ANALYSIS)
ctest_test(BUILD "${CTEST_BINARY_DIRECTORY}"
PARALLEL_LEVEL ${NCPUS}
SCHEDULE_RANDOM ON
RETURN_VALUE _ctest_test_ret_val)
ctest_submit()
endif()
if(_ctest_test_ret_val)
Message(FATAL_ERROR "Some tests failed.")
endif()

118
Jenkinsfile vendored
View File

@@ -1,118 +0,0 @@
#!groovy
def jobMatrix(String type, List specs) {
def nodes = [:]
for (spec in specs) {
def job = ""
def selector = "slurm"
def os = ""
def ver = ""
if (type == 'build') {
job = "${spec.os}-${spec.ver}-${spec.arch}-${spec.compiler}"
if (spec.os =~ /^macos/) {
selector = "${spec.os}-${spec.ver}-${spec.arch}"
}
os = spec.os
ver = spec.ver
} else { // == 'check'
job = "${spec.name}"
os = 'fedora'
ver = '36'
}
def label = "${job}"
def extra = spec.extra
nodes[label] = {
node(selector) {
githubNotify(context: "${label}", description: 'Building ...', status: 'PENDING')
try {
deleteDir()
checkout scm
def jobscript = 'job.sh'
def ctestcmd = "ctest ${extra} -S FairMQTest.cmake -V --output-on-failure"
sh "echo \"set -e\" >> ${jobscript}"
sh "echo \"export LABEL=\\\"\${JOB_BASE_NAME} ${label}\\\"\" >> ${jobscript}"
if (selector =~ /^macos/) {
sh """\
echo \"${ctestcmd}\" >> ${jobscript}
"""
sh "cat ${jobscript}"
sh "bash ${jobscript}"
} else { // selector == "slurm"
def imageurl = "oras://ghcr.io/fairrootgroup/fairmq-dev/${os}-${ver}-sif:latest"
def execopts = "--ipc --uts --pid -B/shared"
def containercmd = "singularity exec ${execopts} ${imageurl} bash -l -c \\\"${ctestcmd} ${extra}\\\""
sh """\
echo \"echo \\\"*** Job started at .......: \\\$(date -R)\\\"\" >> ${jobscript}
echo \"echo \\\"*** Job ID ...............: \\\${SLURM_JOB_ID}\\\"\" >> ${jobscript}
echo \"echo \\\"*** Compute node .........: \\\$(hostname -f)\\\"\" >> ${jobscript}
echo \"unset http_proxy\" >> ${jobscript}
echo \"unset HTTP_PROXY\" >> ${jobscript}
echo \"${containercmd}\" >> ${jobscript}
"""
sh "cat ${jobscript}"
sh "test/ci/slurm-submit.sh \"FairMQ \${JOB_BASE_NAME} ${label}\" ${jobscript}"
if (job == "static-analyzers") {
recordIssues(enabledForFailure: true,
tools: [gcc(pattern: 'build/Testing/Temporary/*.log')],
filters: [excludeFile('extern/*'), excludeFile('usr/*')],
skipBlames: true,
skipPublishingChecks: true)
}
}
deleteDir()
githubNotify(context: "${label}", description: 'Success', status: 'SUCCESS')
} catch (e) {
deleteDir()
githubNotify(context: "${label}", description: 'Error', status: 'ERROR')
throw e
}
}
}
}
return nodes
}
pipeline{
agent none
stages {
stage("CI") {
steps{
script {
def builds = jobMatrix('build', [
[os: 'ubuntu', ver: '20.04', arch: 'x86_64', compiler: 'gcc-9'],
[os: 'ubuntu', ver: '22.04', arch: 'x86_64', compiler: 'gcc-11'],
[os: 'ubuntu', ver: '24.04', arch: 'x86_64', compiler: 'gcc-13'],
[os: 'fedora', ver: '33', arch: 'x86_64', compiler: 'gcc-10'],
[os: 'fedora', ver: '34', arch: 'x86_64', compiler: 'gcc-11'],
[os: 'fedora', ver: '35', arch: 'x86_64', compiler: 'gcc-11'],
[os: 'fedora', ver: '36', arch: 'x86_64', compiler: 'gcc-12'],
[os: 'fedora', ver: '37', arch: 'x86_64', compiler: 'gcc-12'],
[os: 'fedora', ver: '38', arch: 'x86_64', compiler: 'gcc-13'],
[os: 'fedora', ver: '39', arch: 'x86_64', compiler: 'gcc-13'],
[os: 'fedora', ver: '40', arch: 'x86_64', compiler: 'gcc-14'],
[os: 'macos', ver: '14', arch: 'x86_64', compiler: 'apple-clang-16'],
[os: 'macos', ver: '15', arch: 'x86_64', compiler: 'apple-clang-16'],
[os: 'macos', ver: '15', arch: 'arm64', compiler: 'apple-clang-16'],
])
def all_debug = "-DCMAKE_BUILD_TYPE=Debug"
def checks = jobMatrix('check', [
[name: 'static-analyzers', extra: "${all_debug} -DRUN_STATIC_ANALYSIS=ON"],
[name: '{address,leak,ub}-sanitizers',
extra: "${all_debug} -DENABLE_SANITIZER_ADDRESS=ON -DENABLE_SANITIZER_LEAK=ON -DENABLE_SANITIZER_UNDEFINED_BEHAVIOUR=ON -DCMAKE_CXX_FLAGS='-O1 -fno-omit-frame-pointer'"],
[name: 'thread-sanitizer', extra: "${all_debug} -DENABLE_SANITIZER_THREAD=ON -DCMAKE_CXX_COMPILER=clang++"],
])
parallel(builds + checks)
}
}
}
}
}

View File

@@ -68,6 +68,7 @@ function(build_bundled package bundle)
exec(${CMAKE_COMMAND} -S ${${package}_SOURCE_DIR} -B ${${package}_BINARY_DIR} -G ${CMAKE_GENERATOR}
-DCMAKE_INSTALL_PREFIX=${${package}_INSTALL_DIR} -DBUILD_GMOCK=OFF
-DCMAKE_POSITION_INDEPENDENT_CODE=ON
)
exec(${CMAKE_COMMAND} --build ${${package}_BINARY_DIR})
exec(${CMAKE_COMMAND} --build ${${package}_BINARY_DIR} --target install)

View File

@@ -23,6 +23,18 @@ if(BUILD_FAIRMQ OR BUILD_TIDY_TOOL)
find_package2(PUBLIC Boost REQUIRED VERSION 1.66
COMPONENTS container program_options filesystem date_time regex
)
# Check Boost.Process compatibility
# Boost 1.88 has broken Boost.Process v2 without v1 compatibility headers
# Boost 1.89+ provides <boost/process/v1.hpp> for the old API
if(Boost_VERSION VERSION_EQUAL "1.88.0")
message(FATAL_ERROR "Boost version 1.88 is not supported due to Boost.Process API changes. "
"Please use Boost < 1.88 or >= 1.89")
endif()
if(Boost_VERSION VERSION_GREATER_EQUAL "1.89")
set(FAIRMQ_BOOST_PROCESS_V1_HEADER ON CACHE INTERNAL "Use boost/process/v1.hpp for Boost >= 1.89")
endif()
endif()
if(BUILD_FAIRMQ)

View File

@@ -54,8 +54,8 @@ endif()
list(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/${PROJECT_INSTALL_LIBDIR}" isSystemDir)
if("${isSystemDir}" STREQUAL "-1")
if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
list(APPEND CMAKE_EXE_LINKER_FLAGS "-Wl,--enable-new-dtags")
list(APPEND CMAKE_SHARED_LINKER_FLAGS "-Wl,--enable-new-dtags")
string(APPEND CMAKE_EXE_LINKER_FLAGS " -Wl,--enable-new-dtags")
string(APPEND CMAKE_SHARED_LINKER_FLAGS " -Wl,--enable-new-dtags")
list(PREPEND CMAKE_INSTALL_RPATH "$ORIGIN/../${PROJECT_INSTALL_LIBDIR}")
elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
list(PREPEND CMAKE_INSTALL_RPATH "@loader_path/../${PROJECT_INSTALL_LIBDIR}")

View File

@@ -12,6 +12,22 @@ set(test_script_prefix "test-ex")
set(testsuite "Example")
set(transports "zeromq" "shmem")
# Environment for every example test (directory scope, so the subdirectories
# see it too)
set(env_mods)
if(ENABLE_SANITIZER_LEAK AND CMAKE_VERSION VERSION_GREATER_EQUAL 3.22)
get_filename_component(lsan_supps "${CMAKE_SOURCE_DIR}/test/leak_sanitizer_suppressions.txt" ABSOLUTE)
list(APPEND env_mods "LSAN_OPTIONS=set:suppressions=${lsan_supps}")
endif()
# Run the example tests against an alternative runtime library directory as
# well (see test/CMakeLists.txt). The test scripts only launch instrumented
# binaries (bash itself does not link libstdc++). Note: unlike the
# testsuites, the example binaries get no locale-cache warmup -- their
# main() comes from the installed public header fairmq/runDevice.h.
if(FAIRMQ_TEST_LD_LIBRARY_PATH)
list(APPEND env_mods "LD_LIBRARY_PATH=path_list_prepend:${FAIRMQ_TEST_LD_LIBRARY_PATH}")
endif()
function(add_example)
cmake_parse_arguments(PARSE_ARGV 0 ARG
"CONFIG;NO_TRANSPORT;NO_TEST"
@@ -29,11 +45,6 @@ function(add_example)
message(FATAL_ERROR "NAME arg is required")
endif()
if(ENABLE_SANITIZER_LEAK AND CMAKE_VERSION VERSION_GREATER_EQUAL 3.22)
get_filename_component(lsan_supps "${CMAKE_SOURCE_DIR}/test/leak_sanitizer_suppressions.txt" ABSOLUTE)
set(lsan_options "LSAN_OPTIONS=set:suppressions=${lsan_supps}")
endif()
if(ARG_DEVICE)
set(exe_targets)
foreach(device IN LISTS ARG_DEVICE)
@@ -78,8 +89,8 @@ function(add_example)
set(test "${testsuite}.${name}.${transport}")
add_test(NAME ${test} COMMAND ${CMAKE_CURRENT_BINARY_DIR}/${test_script} ${transport})
set_tests_properties(${test} PROPERTIES TIMEOUT "30")
if(lsan_options)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION ${lsan_options})
if(env_mods)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION "${env_mods}")
endif()
else()
foreach(transport IN LISTS transports)
@@ -88,16 +99,16 @@ function(add_example)
set(test "${testsuite}.${name}.${variant}.${transport}")
add_test(NAME ${test} COMMAND ${CMAKE_CURRENT_BINARY_DIR}/${test_script} ${transport} ${variant})
set_tests_properties(${test} PROPERTIES TIMEOUT "30")
if(lsan_options)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION ${lsan_options})
if(env_mods)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION "${env_mods}")
endif()
endforeach()
else()
set(test "${testsuite}.${name}.${transport}")
add_test(NAME ${test} COMMAND ${CMAKE_CURRENT_BINARY_DIR}/${test_script} ${transport})
set_tests_properties(${test} PROPERTIES TIMEOUT "30")
if(lsan_options)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION ${lsan_options})
if(env_mods)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION "${env_mods}")
endif()
endif()
endforeach()

View File

@@ -15,6 +15,8 @@ set_target_properties(${exe} PROPERTIES ENABLE_EXPORTS ON)
set(test "${testsuite}.${name}")
add_test(NAME ${test} COMMAND ${CMAKE_CURRENT_BINARY_DIR}/${exe})
set_tests_properties(${test} PROPERTIES TIMEOUT 30)
if(lsan_options)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION ${lsan_options})
# (the previous `if(lsan_options)` here could never fire -- that variable
# only ever existed in the add_example() function scope)
if(env_mods)
set_tests_properties(${test} PROPERTIES ENVIRONMENT_MODIFICATION "${env_mods}")
endif()

View File

@@ -69,9 +69,15 @@ if(BUILD_FAIRMQ)
runDevice.h
runFairMQDevice.h
shmem/Common.h
shmem/Manager.h
shmem/Message.h
shmem/Monitor.h
shmem/Poller.h
shmem/Segment.h
shmem/Socket.h
shmem/TransportFactory.h
shmem/UnmanagedRegion.h
shmem/UnmanagedRegionImpl.h
tools/Compiler.h
tools/CppSTL.h
tools/Exceptions.h
@@ -96,12 +102,6 @@ if(BUILD_FAIRMQ)
plugins/Builtin.h
plugins/config/Config.h
plugins/control/Control.h
shmem/Message.h
shmem/Poller.h
shmem/UnmanagedRegionImpl.h
shmem/Socket.h
shmem/TransportFactory.h
shmem/Manager.h
zeromq/Common.h
zeromq/Context.h
zeromq/Message.h
@@ -176,6 +176,9 @@ if(BUILD_FAIRMQ)
FAIRMQ_HAS_STD_FILESYSTEM=${FAIRMQ_HAS_STD_FILESYSTEM}
FAIRMQ_HAS_STD_PMR=${FAIRMQ_HAS_STD_PMR}
)
if(FAIRMQ_BOOST_PROCESS_V1_HEADER)
target_compile_definitions(${target} PRIVATE FAIRMQ_BOOST_PROCESS_V1_HEADER)
endif()
if(DEFINED FAIRMQ_CHANNEL_DEFAULT_AUTOBIND)
# translate CMake boolean (TRUE, FALSE, 0, 1, OFF, ON) into C++ boolean literal (true, false)
if(FAIRMQ_CHANNEL_DEFAULT_AUTOBIND)

View File

@@ -181,7 +181,8 @@ try {
// validate channel name
smatch m;
if (regex_search(fName, m, regex("[^a-zA-Z0-9\\-_\\[\\]#]"))) {
static regex const invalidName(R"([^a-zA-Z0-9\-_\[\]#])");
if (regex_search(fName, m, invalidName)) {
ss << "INVALID";
LOG(debug) << ss.str();
LOG(error) << "channel name contains illegal character: '" << m.str(0) << "', allowed characters are: a-z, A-Z, 0-9, -, _, [, ], #";

View File

@@ -513,7 +513,8 @@ void Device::HandleMultipleTransportInput()
fMultitransportProceed = true;
for (const auto& i : fMultitransportInputs) {
threads.emplace_back(thread(&Device::PollForTransport, this, fTransports.at(i.first).get(), i.second));
threads.emplace_back(
&Device::PollForTransport, this, fTransports.at(i.first).get(), i.second);
}
for (thread& t : threads) {

View File

@@ -66,10 +66,13 @@ bool DeviceRunner::HandleGeneralOptions(const fair::mq::ProgOptions& config, boo
if (printLogo) {
LOG(info) << endl
<< " ______ _ _______ _________ " << endl
<< " / ____/___ _(_)_______ |/ /_ __ \\ version " << FAIRMQ_GIT_VERSION << endl
<< " / /_ / __ `/ / ___/__ /|_/ /_ / / / build " << FAIRMQ_BUILD_TYPE << endl
<< " / ____/___ _(_)_______ |/ /_ __ \\ version "
<< FAIRMQ_GIT_VERSION << endl
<< " / /_ / __ `/ / ___/__ /|_/ /_ / / / build " << FAIRMQ_BUILD_TYPE
<< endl
<< " / __/ / /_/ / / / _ / / / / /_/ / " << FAIRMQ_REPO_URL << endl
<< " /_/ \\__,_/_/_/ /_/ /_/ \\___\\_\\ " << FAIRMQ_LICENSE << " © " << FAIRMQ_COPYRIGHT << endl;
<< R"( /_/ \__,_/_/_/ /_/ /_/ \___\_\ )" << FAIRMQ_LICENSE
<< " © " << FAIRMQ_COPYRIGHT << endl;
}
}

View File

@@ -17,7 +17,7 @@
#include <boost/container/container_fwd.hpp>
#include <boost/container/flat_map.hpp>
#include <boost/container/pmr/memory_resource.hpp>
#include <memory_resource>
#include <cstring>
#include <fairmq/Message.h>
#include <stdexcept>
@@ -27,7 +27,7 @@ namespace fair::mq {
class TransportFactory;
using byte = unsigned char;
namespace pmr = boost::container::pmr;
namespace pmr = std::pmr;
/// All FairMQ related memory resources need to inherit from this interface
/// class for the

View File

@@ -87,7 +87,7 @@ struct Parts
const_iterator end() const noexcept { return fParts.end(); }
const_iterator cend() const noexcept { return fParts.cend(); }
container fParts{};
container fParts;
};
} // namespace fair::mq

View File

@@ -32,12 +32,9 @@ const std::string fair::mq::PluginManager::fgkLibPrefixAlt = "FairMQPlugin_";
std::vector<boost::dll::shared_library> fair::mq::PluginManager::fgDLLKeepAlive =
std::vector<boost::dll::shared_library>();
fair::mq::PluginManager::PluginManager()
: fPluginServices()
{}
fair::mq::PluginManager::PluginManager() {}
fair::mq::PluginManager::PluginManager(const vector<string>& args)
: fPluginServices()
{
// Parse command line options
auto options = ProgramOptions();

View File

@@ -17,7 +17,9 @@ auto PluginServices::ChangeDeviceState(const string& controller, const DeviceSta
{
lock_guard<mutex> lock{fDeviceControllerMutex};
if (!fDeviceController) fDeviceController = controller;
if (!fDeviceController) {
fDeviceController = controller;
}
if (fDeviceController == controller) {
return fDevice.ChangeState(next);

View File

@@ -189,7 +189,7 @@ vector<string> ProgOptions::GetPropertyKeys() const
vector<string> keys;
for (const auto& it : fVarMap) {
keys.push_back(it.first.c_str());
keys.emplace_back(it.first.c_str());
}
return keys;

View File

@@ -79,7 +79,7 @@ Properties SuboptParser(const vector<string>& channelConfig, const string& devic
ptree channelsArray;
for (auto token : channelConfig) {
for (const auto& token : channelConfig) {
string channelName;
ptree channelProperties;

View File

@@ -135,7 +135,7 @@ struct RegionConfig
int64_t userFlags = 0; /// custom flags that have no effect on the transport, but can be retrieved from the region by the user
uint64_t size = 0; /// region size
uint64_t rcSegmentSize = 100000000; /// size of the segment that stores reference counts when "soft"-copying the messages
std::string path = ""; /// file path, if the region is backed by a file
std::string path; /// file path, if the region is backed by a file
std::optional<uint16_t> id = std::nullopt; /// region id
uint32_t linger = 100; /// delay in ms before region destruction to collect outstanding events
};

View File

@@ -180,7 +180,6 @@ struct RegionInfo
, fUserFlags(userFlags)
, fSize(size)
, fRCSegmentSize(rcSegmentSize)
, fDestroyed(false)
{}
Str fPath;
@@ -188,7 +187,7 @@ struct RegionInfo
uint64_t fUserFlags;
uint64_t fSize;
uint64_t fRCSegmentSize;
bool fDestroyed;
bool fDestroyed{false};
};
using Uint16RegionInfoPairAlloc = boost::interprocess::allocator<std::pair<const uint16_t, RegionInfo>, SegmentManager>;
@@ -340,4 +339,13 @@ struct SegmentBufferShrink
} // namespace fair::mq::shmem
namespace fair::mq { class TransportFactory; }
namespace fair::mq::shmem {
// Resolve a MetaHeader (received over a side channel) to the local data pointer.
// The caller is responsible for ensuring the backing buffer remains alive for the
// duration of access; FairMQ provides no refcount protection for this path.
char* GetDataAddressFromHandle(fair::mq::TransportFactory& factory, const MetaHeader& meta);
} // namespace fair::mq::shmem
#endif /* FAIR_MQ_SHMEM_COMMON_H_ */

View File

@@ -7,11 +7,18 @@
********************************************************************************/
#include "Manager.h"
#include "TransportFactory.h"
// Needed to compile-firewall the <boost/process/async.hpp> header because it
// interferes with the <asio/buffer.hpp> header. So, let's factor
// the whole dependency to Boost.Process out of the header.
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
#include <boost/process/v1.hpp>
namespace bp = boost::process::v1;
#else
#include <boost/process.hpp>
namespace bp = boost::process;
#endif
#include <fairlogger/Logger.h>
namespace fair::mq::shmem {
@@ -28,7 +35,7 @@ bool Manager::SpawnShmMonitor(const std::string& id)
path.emplace(path.begin(), env.at(fairmq_path_key).to_string());
}
auto exe(boost::process::search_path(shmmonitor_exe_name, path));
auto exe(bp::search_path(shmmonitor_exe_name, path));
if (exe.empty()) {
LOG(warn) << "could not find " << shmmonitor_exe_name << " in \"$" << fairmq_path_key
<< ":$PATH\"";
@@ -39,10 +46,18 @@ bool Manager::SpawnShmMonitor(const std::string& id)
bool verbose(env.count(shmmonitor_verbose_key)
&& env.at(shmmonitor_verbose_key).to_string() == "true");
boost::process::spawn(
bp::spawn(
exe, "-x", "-m", "--shmid", id, "-d", "-t", "2000", (verbose ? "--verbose" : ""), env);
return true;
}
char* GetDataAddressFromHandle(fair::mq::TransportFactory& factory, const MetaHeader& meta)
{
if (factory.GetType() != fair::mq::Transport::SHM) {
throw SharedMemoryError("GetDataAddressFromHandle called on a non-shmem transport");
}
return static_cast<TransportFactory&>(factory).GetDataAddressFromHandle(meta);
}
} // namespace fair::mq::shmem

View File

@@ -390,8 +390,10 @@ class Manager
}
auto* lRegion = GetRegion(id);
fTlRegionCache.fRegionsTLCache.emplace_back(std::make_tuple(lRegion, id, fShmId64));
if (lRegion) {
fTlRegionCache.fRegionsTLCache.emplace_back(lRegion, id, fShmId64);
fTlRegionCache.fRegionsTLCacheGen = fRegionsGen;
}
return lRegion;
}
@@ -776,6 +778,30 @@ class Manager
auto GetMetadataMsgSize() const noexcept { return fMetadataMsgSize; }
// Resolve a MetaHeader (received over a side channel) to the local data pointer.
// The caller is responsible for ensuring the backing buffer remains alive for the
// duration of access; FairMQ provides no refcount protection for this path.
char* GetDataAddressFromHandle(const MetaHeader& meta)
{
if (meta.fManaged) {
if (meta.fSize == 0) {
return nullptr;
}
GetSegment(meta.fSegmentId);
auto it = fSegments.find(meta.fSegmentId);
if (it == fSegments.end()) {
throw SharedMemoryError(tools::ToString("GetDataAddressFromHandle: cannot open segment with id ", meta.fSegmentId));
}
return ShmHeader::UserPtr(GetAddressFromHandle(meta.fHandle, meta.fSegmentId));
} else {
UnmanagedRegion* region = GetRegionFromCache(meta.fRegionId);
if (!region) {
throw SharedMemoryError(tools::ToString("GetDataAddressFromHandle: cannot get unmanaged region with id ", meta.fRegionId));
}
return reinterpret_cast<char*>(region->GetData()) + meta.fHandle;
}
}
~Manager()
{
fRegionsGen += 1; // signal TL cache invalidation

View File

@@ -77,7 +77,12 @@ class Message final : public fair::mq::Message
fManager.IncrementMsgCounter();
}
Message(Manager& manager, UnmanagedRegionPtr& region, void* data, const size_t size, void* hint = 0, fair::mq::TransportFactory* factory = nullptr)
Message(Manager& manager,
UnmanagedRegionPtr& region,
void* data,
const size_t size,
void* hint = nullptr,
fair::mq::TransportFactory* factory = nullptr)
: fair::mq::Message(factory)
, fManager(manager)
, fLocalPtr(static_cast<char*>(data))
@@ -162,6 +167,11 @@ class Message final : public fair::mq::Message
}
}
MetaHeader GetMeta() const
{
return {fSize, fHint, fHandle, fShared, fRegionId, fSegmentId, fManaged};
}
void* GetData() const override
{
if (!fLocalPtr) {

View File

@@ -59,12 +59,18 @@ void signalHandler(int signal)
gSignalStatus = signal;
}
Monitor::Monitor(string shmId, bool selfDestruct, bool interactive, bool viewOnly, unsigned int timeoutInMS, unsigned int intervalInMS, bool monitor, bool cleanOnExit)
Monitor::Monitor(string shmId,
bool selfDestruct,
bool interactive,
bool viewOnly,
unsigned int timeoutInMS,
unsigned int intervalInMS,
bool monitor,
bool cleanOnExit)
: fSelfDestruct(selfDestruct)
, fInteractive(interactive)
, fViewOnly(viewOnly)
, fMonitor(monitor)
, fSeenOnce(false)
, fCleanOnExit(cleanOnExit)
, fTimeoutInMS(timeoutInMS)
, fIntervalInMS(intervalInMS)

View File

@@ -170,7 +170,7 @@ class Monitor
bool fInteractive; // running in interactive mode
bool fViewOnly; // view only mode
bool fMonitor;
bool fSeenOnce; // true is segment has been opened successfully at least once
bool fSeenOnce{false}; // true is segment has been opened successfully at least once
bool fCleanOnExit;
unsigned int fTimeoutInMS;
unsigned int fIntervalInMS;

View File

@@ -71,7 +71,7 @@ class Poller final : public fair::mq::Poller
try {
int offset = 0;
// calculate offsets and the total size of the poll item set
for (std::string channel : channelList) {
for (const std::string& channel : channelList) {
fOffsetMap[channel] = offset;
offset += channelsMap.at(channel).size();
fNumItems += channelsMap.at(channel).size();
@@ -80,7 +80,7 @@ class Poller final : public fair::mq::Poller
fItems = new zmq_pollitem_t[fNumItems];
int index = 0;
for (std::string channel : channelList) {
for (const std::string& channel : channelList) {
for (unsigned int i = 0; i < channelsMap.at(channel).size(); ++i) {
index = fOffsetMap[channel] + i;

View File

@@ -39,18 +39,19 @@ namespace fair::mq::shmem
class Socket final : public fair::mq::Socket
{
public:
Socket(Manager& manager, const std::string& type, const std::string& name, const std::string& id, void* context, fair::mq::TransportFactory* fac = nullptr)
Socket(Manager& manager,
const std::string& type,
const std::string& name,
const std::string& id,
void* context,
fair::mq::TransportFactory* fac = nullptr)
: fair::mq::Socket(fac)
, fManager(manager)
, fId(id + "." + name + "." + type)
, fSocket(nullptr)
, fMonitorSocket(nullptr)
, fBytesTx(0)
, fBytesRx(0)
, fMessagesTx(0)
, fMessagesRx(0)
, fTimeout(100)
, fConnectedPeersCount(0)
, fMetadataMsgSize(manager.GetMetadataMsgSize())
{
assert(context);
@@ -456,15 +457,15 @@ class Socket final : public fair::mq::Socket
private:
Manager& fManager;
std::string fId;
void* fSocket;
void* fMonitorSocket;
void* fSocket{nullptr};
void* fMonitorSocket{nullptr};
std::atomic<unsigned long> fBytesTx;
std::atomic<unsigned long> fBytesRx;
std::atomic<unsigned long> fMessagesTx;
std::atomic<unsigned long> fMessagesRx;
int fTimeout;
mutable unsigned long fConnectedPeersCount;
int fTimeout{100};
mutable unsigned long fConnectedPeersCount{0};
std::size_t fMetadataMsgSize;
};

View File

@@ -118,7 +118,10 @@ class TransportFactory final : public fair::mq::TransportFactory
return std::make_unique<Message>(*fManager, data, size, ffn, hint, this);
}
MessagePtr CreateMessage(UnmanagedRegionPtr& region, void* data, size_t size, void* hint = 0) override
MessagePtr CreateMessage(UnmanagedRegionPtr& region,
void* data,
size_t size,
void* hint = nullptr) override
{
return std::make_unique<Message>(*fManager, region, data, size, hint, this);
}
@@ -198,6 +201,8 @@ class TransportFactory final : public fair::mq::TransportFactory
void Resume() override { fManager->Resume(); }
void Reset() override { fManager->Reset(); }
char* GetDataAddressFromHandle(const MetaHeader& meta) { return fManager->GetDataAddressFromHandle(meta); }
~TransportFactory() override
{
LOG(debug) << "Destroying Shared Memory transport...";

View File

@@ -63,9 +63,7 @@ struct UnmanagedRegion
, fName(MakeShmName(shmId, "rg", cfg.id.value()))
, fQueueName(MakeShmName(shmId, "rgq", cfg.id.value()))
, fRefCountSegmentName(MakeShmName(shmId, "rrc", cfg.id.value()))
, fShmemObject()
, fFile(nullptr)
, fFileMapping()
, fRcSegmentSize(cfg.rcSegmentSize)
, fQueue(nullptr)
, fCallback(nullptr)
@@ -102,7 +100,7 @@ struct UnmanagedRegion
}
fFileMapping = file_mapping(fName.c_str(), read_write);
LOG(debug) << "UnmanagedRegion(): initialized file: " << fName;
fRegion = mapped_region(fFileMapping, read_write, 0, size, 0, cfg.creationFlags);
fRegion = mapped_region(fFileMapping, read_write, 0, size, nullptr, cfg.creationFlags);
} else {
try {
// if opening fails, create
@@ -126,7 +124,7 @@ struct UnmanagedRegion
}
try {
fRegion = mapped_region(fShmemObject, read_write, 0, 0, 0, cfg.creationFlags);
fRegion = mapped_region(fShmemObject, read_write, 0, 0, nullptr, cfg.creationFlags);
if (size != 0 && size != fRegion.get_size()) {
LOG(error) << "Created/opened region size (" << fRegion.get_size() << ") does not match configured size (" << size << ")";
throw TransportError(tools::ToString("Created/opened region size (", fRegion.get_size(), ") does not match configured size (", size, ")"));

View File

@@ -37,8 +37,6 @@ class UnmanagedRegionImpl final : public fair::mq::UnmanagedRegion
fair::mq::TransportFactory* factory)
: fair::mq::UnmanagedRegion(factory)
, fManager(manager)
, fRegion(nullptr)
, fRegionId(0)
{
auto [regionPtr, regionId] = fManager.CreateRegion(size, callback, bulkCallback, std::move(cfg));
fRegion = regionPtr;
@@ -62,8 +60,8 @@ class UnmanagedRegionImpl final : public fair::mq::UnmanagedRegion
private:
Manager& fManager;
shmem::UnmanagedRegion* fRegion;
uint16_t fRegionId;
shmem::UnmanagedRegion* fRegion{nullptr};
uint16_t fRegionId{0};
};
} // namespace fair::mq::shmem

View File

@@ -99,7 +99,7 @@ string getDefaultRouteNetworkInterface()
string interfaceName;
#ifdef __APPLE__ // MacOS
unique_ptr<FILE, decltype(pclose)*> file(
unique_ptr<FILE, int (*)(FILE*)> file(
popen("route -n get default | grep interface | cut -d \":\" -f 2", "r"), pclose);
#else // Linux
ifstream is("/proc/net/route");
@@ -128,7 +128,7 @@ string getDefaultRouteNetworkInterface()
LOG(debug) << "could not get network interface of the default route from /proc/net/route, "
"going to try via 'ip route'";
unique_ptr<FILE, decltype(pclose)*> file(
unique_ptr<FILE, int (*)(FILE*)> file(
popen("ip route | grep default | cut -d \" \" -f 5 | head -n 1", "r"), pclose);
#endif

View File

@@ -10,17 +10,26 @@
#include <fairmq/tools/Strings.h>
#include <boost/asio.hpp>
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
#include <boost/process/v1.hpp>
#else
#include <boost/process.hpp>
#endif
#include <chrono>
#include <csignal> // kill, signals
#include <iostream>
#include <mutex>
#include <sstream>
#include <stdexcept>
#include <thread>
#include <utility>
using namespace std;
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
namespace bp = boost::process::v1;
#else
namespace bp = boost::process;
#endif
namespace ba = boost::asio;
namespace bs = boost::system;
@@ -32,10 +41,18 @@ class LinePrinter
, fPrefix(std::move(prefix))
{}
// prints line with prefix on both cout (thread-safe) and output stream
// prints line with prefix on both cout and output stream
void Print(const string& line)
{
// Serialize: the standard allows concurrent insertion on std::cout,
// but libstdc++ maintains the formatted-output state (e.g.
// ios_base::width) with plain reads and writes that constitute data
// races; the lock also keeps whole lines from interleaving.
static mutex sCoutMutex;
{
lock_guard<mutex> lock(sCoutMutex);
cout << fair::mq::tools::ToString(fPrefix, line, "\n") << flush;
}
fOut << fPrefix << line << endl;
}

View File

@@ -37,7 +37,6 @@ class Context
Context(int numIoThreads)
: fZmqCtx(zmq_ctx_new())
, fInterrupted(false)
, fRegionCounter(1)
{
if (!fZmqCtx) {
throw ContextError(tools::ToString("failed creating context, reason: ", zmq_strerror(errno)));
@@ -180,7 +179,7 @@ class Context
mutable std::mutex fMtx;
std::atomic<bool> fInterrupted;
uint16_t fRegionCounter;
uint16_t fRegionCounter{1};
std::condition_variable fRegionEventsCV;
std::vector<RegionInfo> fRegionInfos;
std::queue<RegionInfo> fRegionEvents;

View File

@@ -107,7 +107,11 @@ class Message final : public fair::mq::Message
}
}
Message(UnmanagedRegionPtr& region, void* data, const size_t size, void* hint = 0, fair::mq::TransportFactory* factory = nullptr)
Message(UnmanagedRegionPtr& region,
void* data,
const size_t size,
void* hint = nullptr,
fair::mq::TransportFactory* factory = nullptr)
: fair::mq::Message(factory)
, fMsg(std::make_unique<zmq_msg_t>())
{

View File

@@ -86,7 +86,7 @@ class Poller final : public fair::mq::Poller
fItems = new zmq_pollitem_t[fNumItems];
int index = 0;
for (std::string channel : channelList) {
for (const std::string& channel : channelList) {
for (unsigned int i = 0; i < channelsMap.at(channel).size(); ++i) {
index = fOffsetMap[channel] + i;

View File

@@ -41,8 +41,7 @@ class Socket final : public fair::mq::Socket
, fBytesRx(0)
, fMessagesTx(0)
, fMessagesRx(0)
, fTimeout(100)
, fConnectedPeersCount(0)
{
if (fSocket == nullptr) {
LOG(error) << "Failed creating socket " << fId << ", reason: " << zmq_strerror(errno);
@@ -405,8 +404,8 @@ class Socket final : public fair::mq::Socket
std::atomic<unsigned long> fMessagesTx;
std::atomic<unsigned long> fMessagesRx;
int fTimeout;
mutable unsigned long fConnectedPeersCount;
int fTimeout{100};
mutable unsigned long fConnectedPeersCount{0};
};
} // namespace fair::mq::zmq

View File

@@ -73,7 +73,10 @@ class TransportFactory final : public fair::mq::TransportFactory
return std::make_unique<Message>(data, size, ffn, hint, this);
}
MessagePtr CreateMessage(UnmanagedRegionPtr& region, void* data, size_t size, void* hint = 0) override
MessagePtr CreateMessage(UnmanagedRegionPtr& region,
void* data,
size_t size,
void* hint = nullptr) override
{
return std::make_unique<Message>(region, data, size, hint, this);
}

View File

@@ -16,18 +16,33 @@ if(FairLogger_VERSION VERSION_LESS 1.9.0 AND FairLogger_VERSION VERSION_GREATER_
LIST(APPEND definitions FAIR_MIN_SEVERITY=trace)
endif()
if(FAIRMQ_BOOST_PROCESS_V1_HEADER)
LIST(APPEND definitions FAIRMQ_BOOST_PROCESS_V1_HEADER)
endif()
if(definitions)
set(definitions DEFINITIONS ${definitions})
endif()
set(test_environment)
if(ENABLE_SANITIZER_LEAK)
get_filename_component(lsan_supps "${CMAKE_CURRENT_SOURCE_DIR}/leak_sanitizer_suppressions.txt" ABSOLUTE)
set(environment ENVIRONMENT "LSAN_OPTIONS=set:suppressions=${lsan_supps}")
list(APPEND test_environment "LSAN_OPTIONS=set:suppressions=${lsan_supps}")
endif()
# Run the tests against an alternative runtime library directory, e.g. a
# tsan-instrumented libstdc++. Set per test (not at the job level), because
# such a library must only be loaded into instrumented executables.
if(FAIRMQ_TEST_LD_LIBRARY_PATH)
list(APPEND test_environment "LD_LIBRARY_PATH=path_list_prepend:${FAIRMQ_TEST_LD_LIBRARY_PATH}")
endif()
if(test_environment)
set(environment ENVIRONMENT ${test_environment})
endif()
add_testhelper(runTestDevice
SOURCES
helper/runTestDevice.cxx
helper/LocaleWarmup.h
helper/devices/TestPairLeft.h
helper/devices/TestPairRight.h
helper/devices/TestPollIn.h
@@ -134,6 +149,7 @@ add_testsuite(Device
${CMAKE_CURRENT_SOURCE_DIR}/device
${CMAKE_CURRENT_BINARY_DIR}
TIMEOUT 20
${definitions}
${environment}
)

View File

@@ -1,19 +0,0 @@
ARG VERSION=latest
FROM fedora:${VERSION}
ARG VERSION=latest
LABEL org.opencontainers.image.source "https://github.com/FairRootGroup/FairMQ"
LABEL org.opencontainers.image.description "FairMQ development environment"
RUN dnf -y update
# https://git.gsi.de/SDE/packages/builder
RUN dnf -y install https://alfa-ci.gsi.de/packages/rpm/fedora-$VERSION-x86_64/fairsoft-release-dev.rpm
RUN dnf -y install clang cli11-devel ninja-build 'dnf-command(builddep)' libasan liblsan libtsan libubsan clang-tools-extra
RUN dnf -y builddep fairmq
RUN dnf -y clean all
# buildah build --build-arg "VERSION=36" -t "ghcr.io/fairrootgroup/fairmq-dev/fedora-36:latest" -f test/ci/Containerfile.fedora .
# echo $GH_PAT | buildah login -u dennisklein --password-stdin ghcr.io
# buildah push ghcr.io/fairrootgroup/fairmq-dev/fedora-36:latest
# apptainer pull docker://ghcr.io/fairrootgroup/fairmq-dev/fedora-36:latest
# echo $GH_PAT | apptainer remote login -u dennisklein --password-stdin oras://ghcr.io
# apptainer push ./fedora-36_latest.sif oras://ghcr.io/fairrootgroup/fairmq-dev/fedora-36-sif:latest

View File

@@ -1,36 +0,0 @@
ARG VERSION=latest
FROM ubuntu:${VERSION}
ARG VERSION=latest
LABEL org.opencontainers.image.source "https://github.com/FairRootGroup/FairMQ"
LABEL org.opencontainers.image.description "FairMQ development environment"
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get -y install ca-certificates patch cmake git libboost-dev libboost-log-dev libboost-system-dev libboost-regex-dev libboost-filesystem-dev libboost-container-dev libboost-thread-dev libboost-date-time-dev libboost-program-options-dev g++ libfmt-dev ninja-build wget libczmq-dev libxml2-utils libfabric-dev libfabric-bin pkg-config
RUN apt-get -y clean
RUN cd /tmp
RUN git clone -b v1.19.2 --recurse-submodules https://github.com/FairRootGroup/asio
RUN cmake -GNinja -S asio -B asio_build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release
RUN cmake --build asio_build --target install
RUN rm -rf asio asio_build
RUN git clone -b v1.0.0 https://github.com/FairRootGroup/FairCMakeModules
RUN cmake -GNinja -S FairCMakeModules -B FairCMakeModules_build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release
RUN cmake --build FairCMakeModules_build --target install
RUN rm -rf FairCMakeModules FairCMakeModules_build
RUN git clone -b v1.11.0 https://github.com/FairRootGroup/FairLogger
RUN cmake -GNinja -S FairLogger -B FairLogger_build -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DUSE_EXTERNAL_FMT=ON
RUN cmake --build FairLogger_build --target install
RUN rm -rf FairLogger FairLogger_build
# buildah build --build-arg "VERSION=22.04" -t "ghcr.io/fairrootgroup/fairmq-dev/ubuntu-22.04:latest" -f test/ci/Containerfile.ubuntu .
# echo $GH_PAT | buildah login -u dennisklein --password-stdin ghcr.io
# buildah push ghcr.io/fairrootgroup/fairmq-dev/ubuntu-22.04:latest
# apptainer pull docker://ghcr.io/fairrootgroup/fairmq-dev/ubuntu-22.04:latest
# echo $GH_PAT | apptainer remote login -u dennisklein --password-stdin oras://ghcr.io
# apptainer push ./ubuntu-22.04_latest.sif oras://ghcr.io/fairrootgroup/fairmq-dev/ubuntu-22.04-sif:latest

View File

@@ -1,42 +0,0 @@
#! /bin/bash
label="$1"
jobsh="$2"
if [ -z "$ALFACI_SLURM_CPUS" ]
then
ALFACI_SLURM_CPUS=20
fi
CPUS_PER_JOB=$(($ALFACI_SLURM_CPUS / 2))
if [ -z "$ALFACI_SLURM_EXTRA_OPTS" ]
then
ALFACI_SLURM_EXTRA_OPTS="--hint=compute_bound"
fi
if [ -z "$ALFACI_SLURM_TIMEOUT" ]
then
ALFACI_SLURM_TIMEOUT=30
fi
if [ -z "$ALFACI_SLURM_QUEUE" ]
then
ALFACI_SLURM_QUEUE=main
fi
echo "*** Slurm request options :"
echo "*** Working directory ..: $PWD"
echo "*** Queue ..............: $ALFACI_SLURM_QUEUE"
echo "*** CPUs ...............: $ALFACI_SLURM_CPUS"
echo "*** Wall Time ..........: $ALFACI_SLURM_TIMEOUT min"
echo "*** Job Name ...........: ${label}"
echo "*** Extra Options ......: ${ALFACI_SLURM_EXTRA_OPTS}"
echo "*** Submitting job at ....: $(date -R)"
(
set -x
srun -p $ALFACI_SLURM_QUEUE -c $CPUS_PER_JOB -n 1 \
-t $ALFACI_SLURM_TIMEOUT \
--job-name="${label}" \
${ALFACI_SLURM_EXTRA_OPTS} \
bash "${jobsh}"
)
retval=$?
echo "*** Exit Code ............: $retval"
exit "$retval"

View File

@@ -0,0 +1,21 @@
spack:
specs:
- "fairlogger@2.2.0 ^fmt@:11"
- "boost@1.87 +container +program_options +filesystem +date_time +regex"
- "libzmq@4.1.4:"
- ninja
- "cmake@3.15:"
concretizer:
targets:
granularity: generic
packages:
all:
require:
- target=x86_64_v3
mirrors:
ghcr-buildcache:
url: oci://ghcr.io/fairrootgroup/fairmq-spack-buildcache
signed: false

21
test/ci/spack-latest.yaml Normal file
View File

@@ -0,0 +1,21 @@
spack:
specs:
- "fairlogger@2.2.0 ^fmt@:11"
- "boost@1.66: +container +program_options +filesystem +date_time +regex"
- "libzmq@4.1.4:"
- ninja
- "cmake@3.15:"
concretizer:
targets:
granularity: generic
packages:
all:
require:
- target=x86_64_v3
mirrors:
ghcr-buildcache:
url: oci://ghcr.io/fairrootgroup/fairmq-spack-buildcache
signed: false

47
test/ci/spack-tsan.yaml Normal file
View File

@@ -0,0 +1,47 @@
spack:
specs:
# fairlogger and boost are left uninstrumented on purpose: no tsan report
# has ever implicated them. Should that change, instrument them with
# per-node flags like libzmq below.
- "fairlogger@2.2.0 ^fmt@:11"
- "boost@1.66: +container +program_options +filesystem +date_time +regex"
# libzmq (and the libsodium it links) are built with ThreadSanitizer
# instrumentation so tsan can observe the happens-before edges that
# libzmq's lock-free queues establish between user and I/O threads.
# Flags are set per node on purpose: the propagating '==' operator could
# reach the gcc node (the compiler is a dependency since spack 1.0) and
# trigger a full compiler rebuild.
- "libzmq@4.1.4: cflags='-g -fno-omit-frame-pointer -fsanitize=thread'
cxxflags='-g -fno-omit-frame-pointer -fsanitize=thread'
ldflags='-fsanitize=thread'
^libsodium cflags='-g -fno-omit-frame-pointer -fsanitize=thread'
ldflags='-fsanitize=thread'"
# ThreadSanitizer-instrumented libstdc++ (from test/ci/spack_repo), a
# drop-in runtime replacement injected per test via LD_LIBRARY_PATH (see
# FAIRMQ_TEST_LD_LIBRARY_PATH in test/CMakeLists.txt).
# Keep the major version in lockstep with the gcc used by the tsan job.
- "libstdcxx-tsan@15"
- ninja
# the per-test env injection uses ctest's ENVIRONMENT_MODIFICATION,
# which needs cmake >= 3.22
- "cmake@3.22:"
concretizer:
targets:
granularity: generic
reuse:
# always re-concretize libstdcxx-tsan from the recipe in
# test/ci/spack_repo: spec reuse would keep resurrecting previously
# built (and cached) variants after the recipe changed; unchanged
# recipes still hit the buildcache because the hash is identical
exclude: ["libstdcxx-tsan"]
packages:
all:
require:
- target=x86_64_v3
mirrors:
ghcr-buildcache:
url: oci://ghcr.io/fairrootgroup/fairmq-spack-buildcache
signed: false

View File

@@ -0,0 +1,90 @@
import os
import shutil
from spack_repo.builtin.build_systems.generic import Package
from spack.package import *
class LibstdcxxTsan(Package):
"""ThreadSanitizer-instrumented build of GCC's libstdc++.
GCC ships no supported way to produce a tsan-instrumented libstdc++ (and
spack's gcc package offers no hook into the target-library flags), so this
package builds only the libstdc++-v3 subtree from the GCC release tarball
matching the toolchain, with -fsanitize=thread. The result is a drop-in
runtime replacement for the compiler's own libstdc++ (same soname and
symbol versions) that lets tsan observe the synchronization inside the
C++ standard library (e.g. std::locale/std::regex lazy initialization).
Select it via LD_LIBRARY_PATH in the environment of the instrumented
executables only. Like any shared library built with gcc
-fsanitize=thread it carries unresolved __tsan_* symbols satisfied by the
executable's libtsan, so loading it into an uninstrumented executable
fails.
Recipe modeled on https://iree.dev/developers/debugging/sanitizers/.
"""
homepage = "https://gcc.gnu.org/onlinedocs/libstdc++/"
url = "https://ftp.gnu.org/gnu/gcc/gcc-15.2.0/gcc-15.2.0.tar.xz"
# Keep in lockstep with the gcc version used by the tsan CI job: a
# libstdc++ older than the compiler may lack GLIBCXX symbol versions
# that binaries built by that compiler reference. Checksums match the
# version() directives in spack's gcc package.
version("15.2.0", sha256="438fd996826b0c82485a29da03a72d71d6e3541a83ec702df4271f6fe025d24e")
version("14.3.0", sha256="e0dc77297625631ac8e50fa92fffefe899a4eb702592da5c32ef04e2293aca3a")
depends_on("c", type="build")
depends_on("cxx", type="build")
depends_on("gmake", type="build")
def install(self, spec, prefix):
# gthr-default.h only materializes in a configured libgcc build dir;
# on POSIX targets it is a verbatim copy of gthr-posix.h. Providing it
# up front spares building libgcc (and the in-tree compiler).
gthr_include = join_path(self.stage.source_path, "spack-gthr")
mkdirp(gthr_include)
copy(
join_path(self.stage.source_path, "libgcc", "gthr-posix.h"),
join_path(gthr_include, "gthr-default.h"),
)
flags = f"-I{gthr_include} -O2 -g -fno-omit-frame-pointer -fsanitize=thread"
build_dir = join_path(self.stage.source_path, "spack-build")
mkdirp(build_dir)
with working_dir(build_dir):
configure = Executable(
join_path(self.stage.source_path, "libstdc++-v3", "configure")
)
configure(
f"--prefix={prefix}",
f"--libdir={prefix.lib}",
"--disable-multilib",
"--disable-libstdcxx-pch",
"--enable-libstdcxx-threads=yes",
"--with-default-libstdcxx-abi=new",
f"CFLAGS={flags}",
f"CXXFLAGS={flags}",
"LDFLAGS=-fsanitize=thread",
)
make()
make("install")
# The runtime libraries land in the multilib os dir (lib64 on
# x86_64), --libdir notwithstanding (--with-toolexeclibdir only
# applies to cross builds); normalize to lib/ for the consumer.
if os.path.isdir(prefix.lib64):
mkdirp(prefix.lib)
for entry in os.listdir(prefix.lib64):
shutil.move(join_path(prefix.lib64, entry), join_path(prefix.lib, entry))
os.rmdir(prefix.lib64)
# Only the runtime library is consumed -- compilation uses the
# compiler's own (identical) headers. Drop the rest to keep the
# buildcache entry small.
for directory in ("include", "share"):
path = join_path(prefix, directory)
if os.path.isdir(path):
shutil.rmtree(path)

View File

@@ -0,0 +1,3 @@
repo:
namespace: fairmq_ci
api: v2.0

View File

@@ -9,7 +9,11 @@
#include "runner.h"
#include <gtest/gtest.h>
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
#include <boost/process/v1.hpp>
#else
#include <boost/process.hpp>
#endif
#include <fairmq/tools/Process.h>
#include <fairmq/tools/Unique.h>
#include <fairmq/Device.h>

View File

@@ -9,7 +9,11 @@
#include "runner.h"
#include <gtest/gtest.h>
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
#include <boost/process/v1.hpp>
#else
#include <boost/process.hpp>
#endif
#include <fairmq/tools/Process.h>
#include <fairmq/tools/Unique.h>

View File

@@ -9,7 +9,11 @@
#include "runner.h"
#include <gtest/gtest.h>
#ifdef FAIRMQ_BOOST_PROCESS_V1_HEADER
#include <boost/process/v1.hpp>
#else
#include <boost/process.hpp>
#endif
#include <fairmq/tools/Unique.h>
#include <fairmq/tools/Process.h>

View File

@@ -29,13 +29,17 @@ TEST(EventManager, Basics)
std::function<void(typename TestEvent::KeyType, int)> callback{
[&](TestEvent::KeyType key, int newValue){
++call_counter;
if (key == "test") value = newValue;
if (key == "test") {
value = newValue;
}
}
};
std::function<void(typename TestEvent::KeyType, string)> callback2{
[&](TestEvent::KeyType key, string newValue){
++call_counter2;
if (key == "test") value2 = newValue;
if (key == "test") {
value2 = newValue;
}
}
};

View File

@@ -0,0 +1,63 @@
/********************************************************************************
* Copyright (C) 2026 GSI Helmholtzzentrum fuer Schwerionenforschung GmbH *
* *
* This software is distributed under the terms of the *
* GNU Lesser General Public Licence (LGPL) version 3, *
* copied verbatim in the file "LICENSE" *
********************************************************************************/
#ifndef FAIR_MQ_TEST_LOCALEWARMUP_H
#define FAIR_MQ_TEST_LOCALEWARMUP_H
#include <locale>
#include <sstream>
namespace fair::mq::test {
/// libstdc++'s std::ctype<char> caches narrow()/widen() results per character
/// in plain char arrays of the global classic-locale facet -- by design with
/// unsynchronized writes, emitted from header-inlined code (locale_facets.h).
/// Whenever two threads exercise an uncached character concurrently (e.g. by
/// compiling a std::regex), ThreadSanitizer rightfully reports the write as a
/// data race. The stores are real and unsynchronized, so instrumenting
/// libstdc++ cannot help; instead, fill the caches before any thread exists,
/// which turns every later access into a pure read.
inline void WarmUpLocaleCaches()
{
auto const& ct = std::use_facet<std::ctype<char>>(std::locale::classic());
for (int c = 1; c < 256; ++c) {
auto const ch = static_cast<char>(c);
ct.narrow(ch, '\0');
ct.widen(ch);
}
// Only the range overload runs _M_narrow_init(), which sets the
// all-cached flag (_M_narrow_ok); the single-char loop above fills the
// cache without it. (widen() needs no equivalent: the single-char
// overload already runs _M_widen_init().)
char from[256];
char to[256];
for (int c = 0; c < 256; ++c) {
from[c] = static_cast<char>(c);
}
ct.narrow(from, from + 256, '?', to);
// num_put/num_get caches used by stream insertion/extraction
std::ostringstream os;
os << 4711 << ' ' << 3.14;
std::istringstream is("42 3.14");
int i = 0;
double d = 0.;
is >> i >> d;
}
/// For binaries that do not own main(): define a namespace-scope instance to
/// run the warm-up during static initialization, before main().
struct LocaleWarmup
{
LocaleWarmup() { WarmUpLocaleCaches(); }
};
} // namespace fair::mq::test
#endif /* FAIR_MQ_TEST_LOCALEWARMUP_H */

View File

@@ -6,6 +6,7 @@
* copied verbatim in the file "LICENSE" *
********************************************************************************/
#include "LocaleWarmup.h"
#include "devices/TestPairLeft.h"
#include "devices/TestPairRight.h"
#include "devices/TestPollIn.h"
@@ -30,6 +31,10 @@
namespace bpo = boost::program_options;
namespace {
[[maybe_unused]] fair::mq::test::LocaleWarmup const gLocaleWarmup{};
}
auto addCustomOptions(bpo::options_description& options) -> void
{
options.add_options()

View File

@@ -1 +1,2 @@
leak:zmq_msg_init_size
leak:zmq::msg_t::init_size

View File

@@ -16,6 +16,7 @@
#include <gtest/gtest.h>
#include <cstring>
#include <string>
#include <vector>
namespace
@@ -101,7 +102,7 @@ TEST(MemoryResources, allocator)
size_t session{tools::UuidHash()};
ProgOptions config;
config.SetProperty<string>("session", to_string(session));
config.SetProperty<std::string>("session", to_string(session));
FactoryType factoryZMQ = TransportFactory::CreateTransportFactory("zeromq", fair::mq::tools::Uuid(), &config);
@@ -129,7 +130,7 @@ TEST(MemoryResources, getMessage)
size_t session{tools::UuidHash()};
ProgOptions config;
config.SetProperty<string>("session", to_string(session));
config.SetProperty<std::string>("session", to_string(session));
config.SetProperty<bool>("shm-monitor", true);
FactoryType factoryZMQ = TransportFactory::CreateTransportFactory("zeromq", fair::mq::tools::Uuid(), &config);

View File

@@ -453,6 +453,46 @@ TEST(EmptyMessage, shmem_expanded_metadata) // NOLINT
EmptyMessage("shmem", "ipc://test_empty_message", true);
}
// GetMeta() + GetDataAddressFromHandle() round-trip: simulates a consumer device that
// receives message metadata over a side channel and resolves it to a local pointer.
// Uses a second factory on the same session to exercise the open_only segment attach path.
auto SideChannel(bool expandedShmMetadata = false) -> void
{
const string session = tools::Uuid();
ProgOptions producerConfig;
producerConfig.SetProperty<string>("session", session);
producerConfig.SetProperty<size_t>("shm-segment-size", 100000000);
producerConfig.SetProperty<bool>("shm-monitor", true);
if (expandedShmMetadata) {
producerConfig.SetProperty<size_t>("shm-metadata-msg-size", 2048);
}
auto producerFactory(TransportFactory::CreateTransportFactory("shmem", tools::Uuid(), &producerConfig));
ProgOptions consumerConfig;
consumerConfig.SetProperty<string>("session", session);
consumerConfig.SetProperty<size_t>("shm-segment-size", 100000000);
consumerConfig.SetProperty<bool>("shm-monitor", true);
auto consumerFactory(TransportFactory::CreateTransportFactory("shmem", tools::Uuid(), &consumerConfig));
const size_t size = 2;
MessagePtr msg(producerFactory->CreateMessage(size));
memcpy(msg->GetData(), "AB", size);
// producer side: extract metadata to send over the side channel
const auto meta = static_cast<const shmem::Message&>(*msg).GetMeta();
EXPECT_EQ(meta.fSize, size);
EXPECT_TRUE(meta.fManaged);
// consumer side: resolve via a different factory — exercises open_only segment attach.
// The virtual address differs between factory mappings; only the content must match.
char* ptr = shmem::GetDataAddressFromHandle(*consumerFactory, meta);
ASSERT_NE(ptr, nullptr);
EXPECT_EQ(ptr[0], 'A');
EXPECT_EQ(ptr[1], 'B');
}
TEST(ZeroCopy, shmem) // NOLINT
{
ZeroCopy();
@@ -463,6 +503,93 @@ TEST(ZeroCopy, shmem_expanded_metadata) // NOLINT
ZeroCopy(true);
}
// Uses a second factory on the same session to exercise the remote region lookup path.
auto SideChannelUnmanaged() -> void
{
const string session = tools::Uuid();
ProgOptions producerConfig;
producerConfig.SetProperty<string>("session", session);
producerConfig.SetProperty<size_t>("shm-segment-size", 100000000);
producerConfig.SetProperty<bool>("shm-monitor", true);
auto producerFactory(TransportFactory::CreateTransportFactory("shmem", tools::Uuid(), &producerConfig));
ProgOptions consumerConfig;
consumerConfig.SetProperty<string>("session", session);
consumerConfig.SetProperty<size_t>("shm-segment-size", 100000000);
consumerConfig.SetProperty<bool>("shm-monitor", true);
auto consumerFactory(TransportFactory::CreateTransportFactory("shmem", tools::Uuid(), &consumerConfig));
const size_t regionSize = 1000000;
tools::Semaphore blocker;
auto region = producerFactory->CreateUnmanagedRegion(regionSize, [&blocker](void*, size_t, void*) {
blocker.Signal();
});
const size_t size = 2;
auto msg(producerFactory->CreateMessage(region, static_cast<char*>(region->GetData()), size, nullptr));
memcpy(msg->GetData(), "AB", size);
const auto meta = static_cast<const shmem::Message&>(*msg).GetMeta();
EXPECT_EQ(meta.fSize, size);
EXPECT_FALSE(meta.fManaged);
// consumer resolves via its own factory — exercises remote region lookup.
// The virtual address differs between factory mappings; only the content must match.
char* ptr = shmem::GetDataAddressFromHandle(*consumerFactory, meta);
ASSERT_NE(ptr, nullptr);
EXPECT_EQ(ptr[0], 'A');
EXPECT_EQ(ptr[1], 'B');
msg.reset();
blocker.Wait();
}
auto SideChannelErrors() -> void
{
ProgOptions config;
config.SetProperty<string>("session", tools::Uuid());
config.SetProperty<size_t>("shm-segment-size", 100000000);
config.SetProperty<bool>("shm-monitor", true);
auto shmFactory(TransportFactory::CreateTransportFactory("shmem", tools::Uuid(), &config));
auto zmqFactory(TransportFactory::CreateTransportFactory("zeromq", tools::Uuid(), &config));
// non-shmem factory must throw
shmem::MetaHeader meta{};
meta.fManaged = true;
EXPECT_THROW(shmem::GetDataAddressFromHandle(*zmqFactory, meta), shmem::SharedMemoryError);
// bad segment id must throw
meta.fSize = 1;
meta.fSegmentId = 999;
EXPECT_THROW(shmem::GetDataAddressFromHandle(*shmFactory, meta), shmem::SharedMemoryError);
// zero-size managed message must return nullptr (mirrors Message::GetData())
meta.fSize = 0;
EXPECT_EQ(shmem::GetDataAddressFromHandle(*shmFactory, meta), nullptr);
}
TEST(SideChannel, shmem) // NOLINT
{
SideChannel();
}
TEST(SideChannel, shmem_expanded_metadata) // NOLINT
{
SideChannel(true);
}
TEST(SideChannel, shmem_unmanaged) // NOLINT
{
SideChannelUnmanaged();
}
TEST(SideChannel, errors) // NOLINT
{
SideChannelErrors();
}
TEST(ZeroCopyFromUnmanaged, shmem) // NOLINT
{
ZeroCopyFromUnmanaged("ipc://test_zerocopy_unmanaged", false, 10000000);

View File

@@ -156,7 +156,7 @@ TEST_F(PluginServices, SubscriptionThreadSafety)
std::array<std::unique_ptr<std::thread>, subscribers> threads;
auto id = 0;
for (auto& thread : threads) {
thread = std::make_unique<std::thread>([&](){
thread = std::make_unique<std::thread>([&, id](){
auto const subscriber = fair::mq::tools::ToString("subscriber_", id);
for (auto i = 0; i < attempts; ++i) {
mServices.SubscribeToDeviceStateChange(subscriber, [](DeviceState){});

View File

@@ -10,6 +10,7 @@
#include <TestEnvironment.h>
#include <gtest/gtest.h>
#include <helper/LocaleWarmup.h>
#include <string>
@@ -24,6 +25,7 @@ string runTestDevice = "@RUN_TEST_DEVICE@";
int main(int argc, char** argv)
{
fair::mq::test::WarmUpLocaleCaches();
::testing::InitGoogleTest(&argc, argv);
::testing::FLAGS_gtest_death_test_style = "threadsafe";
setenv("FAIRMQ_PATH", FAIRMQ_TEST_ENVIRONMENT, 0);