Commit Graph

152 Commits

Author SHA1 Message Date
Alexey Rybalchenko
dbdabd23a4 Zmq: remove global (static) state, refactor 2020-05-18 14:32:19 +02:00
Alexey Rybalchenko
7a67719a3c Remove useless members 2020-05-18 14:32:19 +02:00
Alexey Rybalchenko
d22023bcb5 Implement bulk callbacks for unmanaged regions 2020-05-18 14:32:19 +02:00
Alexey Rybalchenko
e9318dd234 Add FairMQTransportFactory::GetId() 2020-05-11 12:02:19 +02:00
Alexey Rybalchenko
c8fc5ad33f Add bool FairMQTransportFactory::SubscribedToRegionEvents() 2020-05-11 12:02:19 +02:00
Alexey Rybalchenko
a3afadb824 Call region event callback with local_only event for zmq 2020-05-11 12:02:19 +02:00
Alexey Rybalchenko
9992811822 Implement region events for zmq 2020-05-11 12:02:19 +02:00
Alexey Rybalchenko
e1a113aabe Add region events subscriptions 2020-04-28 14:09:04 +02:00
Alexey Rybalchenko
a545bee3b1 Do not report interruption by system call as error 2020-03-02 13:42:37 +01:00
Alexey Rybalchenko
a2cff5b7bb Shmem: simplify message/socket and refactor to use namespaces 2019-12-20 14:04:58 +01:00
Alexey Rybalchenko
a53e95b5f6 Add flatbuffers-based command format to SDK/plugin 2019-09-30 12:18:49 +02:00
Alexey Rybalchenko
5c943bbedb Add debug output 2019-07-11 18:58:46 +02:00
Alexey Rybalchenko
cba6d19781 Add config plugin class. 2019-07-10 19:35:18 +02:00
Alexey Rybalchenko
0e35f1cb22 Shmem region: support huge pages via path to hugetlbfs mount 2019-07-05 15:58:28 +02:00
Alexey Rybalchenko
36f409dc72 Formatting 2019-02-25 12:19:50 +01:00
Alexey Rybalchenko
ee24144d61 Remove previously deprecated Copy method (since 1yr) 2018-11-28 21:16:29 +01:00
Alexey Rybalchenko
a9619a06d0 Add test for FairMQMessage::Rebuild 2018-11-28 21:16:29 +01:00
mkrzewic
cc0c525e0d Set pointer to factory also when receiving multi-part 2018-11-27 14:36:35 +01:00
Alexey Rybalchenko
25fcf13985 Move Bind/Connect/Attach to FairMQChannel 2018-11-13 11:08:48 +01:00
mkrzewic
d40bbfe208 Equip FairMQMessage with pointer to factory (at creation)
the patch seems big but most of it is just propagating the new notion of
constness of the factory - since it keeps track of created messages with
the internal allocator it no longer is const
2018-10-31 13:12:38 +01:00
Alexey Rybalchenko
0cfa9192d7 Deprecate Send-/ReceiveAsync, use timeout variant instead 2018-10-30 10:39:35 +01:00
Alexey Rybalchenko
ce4062f3a0 Remove GetSocket interface that exposes transport details 2018-10-17 13:28:50 +02:00
Alexey Rybalchenko
f8824335a5 Add setters/getters for socket options 2018-10-17 13:28:50 +02:00
Alexey Rybalchenko
dfa1b68867 Make factory classes final (optimization potential) 2018-10-17 13:28:50 +02:00
Alexey Rybalchenko
44acd4997d Implement nanomsg linger in our transport 2018-10-17 13:28:50 +02:00
Alexey Rybalchenko
cfb727181f Remove set/get timeout from general socket interface 2018-10-12 20:29:50 +02:00
Alexey Rybalchenko
e1f555bc05 Fix issues found by Codacy 2018-10-08 17:20:02 +02:00
Alexey Rybalchenko
b814e40c87 Fix multipart transfer timeout and enable its tests 2018-08-27 18:12:02 +02:00
Alexey Rybalchenko
1554c1c273 Change missing options in transport from warning to debug
Transport has meaningful defaults if FairMQProgOptions is missing
2018-06-13 18:03:16 +02:00
Alexey Rybalchenko
e4c349888d Improve compilation speed 2018-05-17 16:32:49 +02:00
Alexey Rybalchenko
7a4fd96b27 Fix namespaces in Transports.h and add conversion map 2018-05-15 19:18:11 +02:00
Alexey Rybalchenko
fed7601be4 replace cmd socket with transport specific unblock mechanism 2018-03-26 13:58:20 +02:00
Dennis Klein
3670dd8835 FairMQ: Remove unused Attach method and unused socket flags 2018-03-26 13:58:20 +02:00
Alexey Rybalchenko
a3393e600e convert log severities use to lowercase and remove use of MQLOG 2018-01-16 22:53:26 +01:00
Alexey Rybalchenko
b59f1b950d FairMQ: Add hint argument to the region callback, settable per message. 2017-12-18 12:53:58 +01:00
Alexey Rybalchenko
ba78964e29 FairMQ: add FairMQMessage::Copy(const FairMQMessage& msg), deprecate the old one. 2017-12-10 17:37:53 +01:00
Alexey Rybalchenko
ea7ae3ded9 remove Get/SetMessage from interface (internal transport detail) 2017-12-10 17:37:53 +01:00
Alexey Rybalchenko
e5aa85b61d FairMQ: Add functionality to set used message size 2017-12-10 17:37:53 +01:00
Mohammad Al-Turany
b5e32403ea correct typo in copyright section 2017-11-29 13:11:35 +01:00
Alexey Rybalchenko
58a312b730 FairMQ: Introduce callbacks for the FairMQUnmanagedRegion.
Callbacks are called when the data buffer of the message assiciated
with the corresponding region is no longer needed by the transport.
Example in examples/advanced/Region/
2017-11-29 09:06:17 +01:00
Alexey Rybalchenko
0bb866ff36 FairMQ: shared memory unmanaged region updates:
- use same mapping for sender/receiver (created/remote) region.
 - rename FairMQRegion -> FairMQUnmanagedRegion.
2017-10-20 14:17:15 +02:00
Alexey Rybalchenko
2db114bc5c Eliminate warnings in FairLink and FairMQ Plugins. 2017-10-05 15:32:12 +02:00
Alexey Rybalchenko
221d2567f0 FairMQRegion and examples/advanced/Region 2017-08-08 14:18:10 +02:00
Dennis Klein
8bc21675af Refactor TransportFactory to RAII
* Remove explicit Initialize and Terminate states,
    map them onto ctor/dtor
  * Remove no longer needed Shutdown state
  * Remove deprecated SetTransport()
2017-05-24 16:37:33 +02:00
Alexey Rybalchenko
6b221d950c Add NewPoller() wrapper. 2017-05-23 08:32:13 +02:00
Dennis Klein
3be2f297f3 generate id if no device id available
* CreateSocket factory no longer accepts id param, but the
    TransportFactory has an id member instead
2017-05-18 20:01:44 +02:00
Dennis Klein
3205e0c378 refactor to more idiomatic RAII
* FairMQTransportFactoryZMQ: move the config invariant
    initialization to ctor
  * FairMQChannel: add new ctor that creates usable channel
  * FairMQSocket*: close sockets in dtor
  * FairMQTransportFactory*: terminate context in dtor
  * FairMQChannel: add Bind/Connect facades (for explicit control, e.g. timing)
2017-05-18 20:01:44 +02:00
Alexey Rybalchenko
0926a9a764 FairMQ: allow accumulation of parts on receive. 2017-04-28 13:30:51 +02:00
Dennis Klein
a1fe478ce7 fix more -Winconsistent-missing-override warnings 2017-04-20 11:07:50 +02:00
Alexey Rybalchenko
c51575f88c fix clang++ -Winconsistend-missing-override warnings 2017-04-20 11:07:50 +02:00
Alexey Rybalchenko
5aaf27bf02 Refactor the transport interface
- give transport Initialize() method with access to device config.
 - avoid using global context in the transport.
 - simplify shutdown procedure (no need for extra thread).
2017-04-20 11:07:49 +02:00
Alexey Rybalchenko
3c73c690f7 Transports: use transportType as static member. 2017-03-14 17:51:26 +01:00
Alexey Rybalchenko
2293c5e417 FairMQChannel: API to access socket metrics; make fSocket private. 2017-03-14 17:51:26 +01:00
Giulio Eulisse
2a72d58766 Remove unneeded if when deleting NULL / 0 / nullptr
C++ standard dictates that NULL / 0 / nullptr is a valid argument for
delete which then simply has no effect:

"The value of the first argument supplied to a deallocation function may
be a null pointer value; if so, and if the deallocation function is one
supplied in the standard library, the call has no effect."

I therefore think in these particular case it's safe to remove the ifs.
2017-03-13 15:46:56 +01:00
Alexey Rybalchenko
c66fd6fe91 Adding multiple transports support & other fixes:
- Avoid polling when only one input channel is used.
 - Send only handles for shared memory transport.
 - Avoid waiting in the rate logger thread when nothing to log.
 - Hide warnings from generated files
 - Fix #483
2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
3e7cb85816 Fix incorrect calculation of transfer rate for multipart. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
8cf1cbb930 Initialize shared memory only if it is used. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
a332d9fc83 First version of the shared memory transport.
Use via `--transport shmem` cmd option. No pub/sub.
2016-12-18 14:50:58 +01:00
Alexey Rybalchenko
b166cedb63 Convert factory methods to return smart ptrs
- Convert factory methods to return smart ptrs.
 - Refactor state machine to use same thread for user states.
 - Remove unused includes and dependencies, use std.
2016-11-18 14:19:16 +01:00
Alexey Rybalchenko
16fd63cd5b Enable new callback API
- OnData() channel data handler.
 - ConditionalRun() for devices without incoming data.
 - Header file with common main(), to be extended with getDevice/addCustomOptions.
 - Update examples (MQ/Tutorial3) to use the new API and config.
 - NewSimpleMessage() for simpler creation of small messages (additional copy).
 - Replace SetProperty/GetProperty with fConfig access.
 - Runtime configurable channel names for common devices.
 - Configurable logging interval per channel.
 - FairMQMultiplier for distributing same data to multiple outputs.
 - Cleanup state machine messages.
 - Cmd option to toggle signal handling.
 - Simpler API for send/receive timeouts.
 - Enable --log-to-file.
 - Fix coverity issues, warnings.
 - Various code cleanup and minor tweaks.
2016-09-30 14:36:35 +02:00
Alexey Rybalchenko
31456e71ff Threads safe socket rate stats 2016-05-12 17:28:32 +02:00
Alexey Rybalchenko
3353e214a7 Add shared memory example
- Add shared memory example in examples/MQ/SharedMemory
 - Device/Task termination: try soft first, and abort if it fails
 - Interactive mode: prevent cin from blocking forever (poll)
2016-05-11 14:52:53 +02:00
Alexey Rybalchenko
a892a5a744 Update FairMQParts with doxygen comments and non-blocking send 2016-04-14 12:51:55 +02:00
Alexey Rybalchenko
82090c356c Fix identity setting for req/rep socket 2016-04-07 15:13:51 +02:00
Alexey Rybalchenko
732373faa2 Update multi-part features (nanomsg) and various fixes
- Implement nanomsg multipart with MessagePack.
 - Use the MessagePack from FairSoft and handle not found case.
 - Update splitter, merger and proxy devices to handle multi-part.
 - Let FairMQParts.At() return pointer reference (can be used for moving).
 - Add missing const specifier in the message interface.
 - Add transmit kernel size setting to channels (ZMQ_SNDBUF).
 - Remove FairMQBuffer device.
 - Remove old multi-part methods from Tutorial3 example (to be replaced with Parts API).
 - Make callback mandatory for newMsg(data, size, callback).
 - Add missing <vector> include in FairMQSocket.
2016-03-24 13:36:47 +01:00
Florian Uhlig
fa7040fe65 Fix compiler warnings.
Initialize all data members in initializer lists.
Reorder data members in initializer list to have the same order as in the class declaration.
Comment or remove unused parameters and unused variables.
Convert all old style casts to the correct and explicit c++ cast like const_cast, static_cast,
dynamic_cast or reinterpret_cast. In most cases static_cast is used.
2016-03-14 13:37:55 +01:00
Alexey Rybalchenko
c42b6ca4ae Include device ID in the zeromq socket identity.
For request sockets in ZeroMQ the socket identity must be unique, otherwise multiple clients will be rejected.
Update the tests to test this use case.
2016-03-03 13:10:56 +01:00
Alexey Rybalchenko
e1fef82657 FairMQ: Extend Multipart and messaging API
- Extend the multipart API to allow sending vectors of messages or helper
   thin wrapper FairMQParts. See example in examples/MQ/8-multipart.
 - NewMessage() can be used in devices instead of
   fTransportFactory->CreateMessage().
   Possible arguments remain unchanged (no args, size or data+size).
 - Send()/Receive() methods can be used in devices instead of
   fChannels.at("chan").at(i).Send()/Receive():
   Send(msg, "chan", i = 0), Receive(msg, "chan", i = 0).
 - Use the new methods in MQ examples and tests.
 - No breaking changes, but FAIRMQ_INTERFACE_VERSION is incremented to 3
   to allow to check for new methods.
2016-02-29 16:25:39 +01:00
Alexey Rybalchenko
f1abb9ecdd Remove compile time transport interface switch
- Remove the compile time check of the transport implementation.
  The transport (zeromq/nanomsg) can be chosen at run time with:
  `device.SetTransport("zeromq"); // possible values are "zeromq" and "nanomsg"`.

  For devices that use FairMQProgOptions, the transport can be configured via cmd option:
  `--transport zeromq` or `--transport nanomsg`. Default values is "zeromq".
  The device receives the configured value with:
  `device.SetTransport(config.GetValue<std::string>("transport"));`

  Old method of setting transport still works. But the NANOMSG constant is not defined.

- Remove old `fairmq/prototest` directory. It was only used as a test for protobuf.
  The protobuf part of Tutorial3 does the same (with different values).

- Fix a bug in FairMQPollerNN, where the `revents` value was not initialized.
  This caused the `poller->CheckOutput()` to trigger when it should not.
2016-01-14 15:19:30 +01:00
Alexey Rybalchenko
e4fed2fa1b Fix copy constructor & assignment operator warning in FairMQChannel
Fix Weffc++ warnings

  - Add missing copy constructors and assignment operators .
  - Hide the warning from FairMQStateMachine.h where it is produced by boost and/or is intended.
  - Some code cleanup.
2016-01-14 14:19:03 +01:00
Florian Uhlig
d254efb91f Fix compiler warnings. Mostly unused parameters have been removed. 2016-01-14 14:19:02 +01:00
Alexey Rybalchenko
8f59db1283 Add methods to set timeout on blocking Send/Receive 2015-10-19 09:52:53 +02:00
Alexey Rybalchenko
0b11ad9274 Fix the type mismatch in the multi-channel poller 2015-10-07 16:46:09 +02:00
Alexey Rybalchenko
19afacb504 FairMQ Examples cleanup
- Rename Tutorial3 MQ files for uniform naming.
  - Add search for dylib in FindDDS.cmake (OSX).
  - Add more detail to the DDS example readme.
  - MQ Example 3 (DDS): choose network interface via command line option.
  - Give FairMQ examples their own CMakeLists.txt for clarity.
  - Remove C++11 checks in Tutorial3 from the code (they are now in CMake).
  - Add Serializer for device properties (FairMQDevice::ListProperties()).
2015-10-05 18:06:55 +02:00
Alexey Rybalchenko
8b71e4d20b Use FairMQDevice::CatchSignals for Tutorial7 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
f13bb5995d Add FairMQ tests (PUB-SUB, PUSH-PULL, REQ-REP). 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
fbf7dbf2ba Add orthogonal OK/ERROR states.
Replace state check mutex with atomic.

Update DDS example documentation.
2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
a7ab33a10e Add new Send/Receive methods with smart pointers and no flag checks. 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
1302e77a16 Several FairMQ fixes and improvements:
- FairMQ: add possibility to poll on multiple channels.
- FairMQ: include command channel when polling on blocking calls (for unblocking without termination).
- FairMQ: move signal handler inside of FairMQDevice class (call FairMQDevice::CatchSignals() in the main function).
- FairMQ: add 'bool CheckCurrentState(statename)' (instead of 'GetCurrentState() == statename' that cannot be thread safe).
- FairMQDevice: add 'InteractiveStateLoop()' method that can be used to change states from the command line.
- FairMQDevice: add automatic transition to IDLE state if Run() exits without an external event.
- FairMQDevice: implement device reset.
- FairMQDevice: use unordered_map for device channels.
- FairMQChannel: improve address validation for channels.
- FairMQChannel: add ExpectsAnotherPart() method to check if another msg part is expected (old approach still works).
- FairMQ: remove invalid transition from the run files.
- FairMQFileSink: disable ROOT termination signal handler.
- Tutorial3: spawn xterm windows from start scripts without overlapping for better visibility.
- FairMQ Examples: update protobuf test and move its files to a common directory.
- FairMQStateMachine: improve feedback on invalid transitions (more readable).
2015-09-28 12:17:22 +02:00
Alexey Rybalchenko
7fda980710 Update FairMQStateMachine & introduce FairMQChannels
Organize sockets as a map of vectors of FairMQChannels.

Update FairMQStateMachine by removing SETTINGINPUT, SETTINGOUTPUT,
BIND and CONNECT states and by adding INITIALIZING_TASK, RESETTING_TASK
and RESETTING_DEVICE states. Run states functions in their own thread.
2015-06-16 09:29:53 +02:00
Alexey Rybalchenko
26a1033e9d Rename device property to have a common format and some code cleanup.
FairMQDevice: Rename property for socket rate logging to have common format (old name is still available for compatibility).
FairMQ: Avoid using std namespace in class headers (may require adding std namespace to some child devices).
FairMQ: A bit of code cleanup
FairMQConfigurable: Stop with an error if a property assignment failed due to incorrect key.
2015-05-28 09:32:30 +02:00
Alexey Rybalchenko
61f24eb73a Increase maximum number of zeromq sockets and improve output on errors.
FairMQSampler: Modified: comment out an unused loop.
FairMQ: Modified: increase maximum number of ZeroMQ sockets (from default 1024).
FairMQ: Modified: reduce amount of output when finding port from a range.
FairMQ: Modified: stop the Device when socket creation fails and output an error.
2015-05-28 09:32:27 +02:00
Alexey Rybalchenko
8a82afe184 Expose BIND and CONNECT states for use with dynamic configuration
introduce FairMQ interface version
2015-02-11 15:15:23 +01:00
Alexey Rybalchenko
6518b7cd41 FairMQSocket: add versions of Send/Receive methods with int flags instead of string, which is more flexible and performant.
Int flags are mapped to their ZeroMQ/nanomsg versions behind the transport interface.
Methods with string flags are kept for backwards compatibility.
2015-01-28 20:01:18 +01:00
Alexey Rybalchenko
ce58ee2302 Fix warnings produced with -Weffc++ in fairmq, base/MQ and Tutorial3.
Remaining warnings originate from boost::msm and boost::mpl.
Also, warnings with missing initializer list entries for DeviceOptions are false positives, since those are PODs.
2015-01-21 16:19:16 +01:00
Alexey Rybalchenko
101bc4c9e7 add flag for non-blocking operations 2014-10-21 17:20:29 +02:00
Alexey Rybalchenko
6968f57abc add ROUTER/DEALER/PAIR sockets. 2014-10-15 11:24:41 +02:00
Alexey Rybalchenko
9317f06c10 Add REQ-REP zmq/nn sockets to FairMQ together with a little example program.
Also includes example of using boost::program_options.
2014-10-10 18:41:36 +02:00
Alexey Rybalchenko
0a610926a1 - Proper process termination:
if interrupted with CTRL+C blocking socket calls will return with -1. Each device should call FairMQDevice::Shutdown() before ending the running state to close open sockets, otherwise the interrupt call itself will block.

- FIX: Update number of received messages for FairMQFileSink.
- Add ability to poll on outputs for FairMQPoller.
2014-08-20 13:48:51 +02:00
Alexey Rybalchenko
8cd120aef4 - Add multipart support to the interface and enable its use out of tasks.
Examples on the use out of tasks are provided in:
  `example/Tutorial3/digitization/TestDetectorDigiLoader.tpl:76-85`: sending a part.
  `example/Tutorial3/reconstruction/FairTestDetectorMQRecoTask.tpl:177-182`: receiving a part.

- This commit also makes structure within processorTask more consistent with samplerTask.

- add macro MQLOG to FairMQLogger.
2014-08-08 14:52:31 +02:00
Alexey Rybalchenko
699671a0f1 Extend the FairMQ transport interface by allowing the user of the FairMQMessage class to define his own deallocation function.
This function will be called when the transport machanism no longer needs the data.

Use this extension with the Protobuf data format, to enable more efficient transport, avoiding memcpy.
2014-06-12 15:31:43 +02:00
Florian Uhlig
865c0e010f Add license file LICENSE with LPGL license text copied from https://www.gnu.org/licenses/lgpl.html. Add license text to most of files of the project.
Add license text to all header files

Add license text to all c++ source files

Add license text to all C macro files

Converting file to UNIX format.

Add license text to all build system files

Add license text to all c source files

Rename file LICENCE to LICENSE. Change preamble in all files.
2014-06-06 14:57:56 +02:00
Alexey Rybalchenko
c03d4ad6d5 use clang-format for Tutorial3 2014-06-06 14:30:41 +02:00
Alexey Rybalchenko
08e603911d TMessage transport implemented! 2014-06-06 14:30:41 +02:00
Alexey Rybalchenko
622e9e9c09 start experimenting with root transport mechanism.
enable efficient message copying for zeromq (currently used only in benchmarkSampler).
cleanup some code.
2014-06-06 14:30:41 +02:00
Alexey Rybalchenko
68d51d8ed5 use clang-format for FairMQ 2014-06-06 14:30:41 +02:00
Alexey Rybalchenko
e80e6d4269 add support for I/O threads to the interface (only used with ZeroMQ). 2014-05-27 19:01:49 +02:00
Alexey Rybalchenko
9a0ed2ca72 Add (optional) Google Protocol Buffers support (example in Tutorial 3).
To use protobuf, run cmake as follows:
cmake -DUSE_PROTOBUF=1 ..
For this, protobuf library has to be installed on the system.

Further changes:
Clean up splitter/merger: default are N-to-1-merger and 1-to-N-splitter.
Fix bug in nanomsg message deallocation.
Setup proper buffer sizes for nanomsg/zeromq via cmake/bash script.
chmod +x for start scripts.
2014-03-17 12:21:47 +01:00
Alexey Rybalchenko
ac8bbf154a simplify logger for shorter calls and readability 2014-02-13 15:55:25 +01:00
NicolasWinckler
dfc1dd6a06 - Make FairMQSampler generic.
- Remove fairroot dependency from fairmq.
2014-02-07 12:36:01 +01:00