=============================== BeckhoffSim Implementation Spec =============================== Karabo Team Introduction ============ BeckhoffSim is a TCP server that simulates a Beckhoff PLC on TCP protocol level. It is used to test BeckhoffCom and BeckhoffDevices without the need to have actual PLC hardware up and running. BeckhoffSim is a Python Karabo device. It is composed of several building blocks, which are implemented as separate classes: the Manager, the TcpServer, a TimeSource, and the various soft devices inheriting from BaseDevice (e.g. DigitalInput). The Manager maintains a set of soft devices, generates the self description, and decodes/encodes TCP messages. The TcpServer opens a TCP port, to which a BeckhoffCom instance can connect, and sends and receives messages. The TimeSource generates train ID and timestamp information. Finally, the soft devices simulate the behaviour of their PLC counterpart. The association between the building blocks in unidirectional, i.e. that method calls are defined only in one direction. Information flow into the opposite direction is realized by events. Blocks interested in a specific event can subscribe to it by passing in a method, which is called if the event is fired. Manager ======= The manager implements the core logic of the BeckhoffSim device. Main task is to maintain a set of simulated soft devices and handle the message transfer between these devices and the TCP server. Each soft device has a set of properties and hardware I/Os, which are exposed by the manager as expected parameters of the BeckhoffSim device. Soft device commands are not exposed, so that they can only be executed via the TCP server. The manager also generates the self description when a BeckhoffCom connects to the TCP server, and sends the heartbeat in regular intervals. The manager is implemented in the class ``Manager`` from module ``core.manager``. It has several dependencies, namely - Framework - TCP server - Time source The dependencies are injected as constructor arguments. Using dependencies is essential for unit testing, since then and only then they can be mocked[#]_. Dependencies are usually defined as interfaces[#]_. Since Python does not need a formal interface contract as e.g. Java or C#, the interface and it's implementation are usually not separated into different classes. .. [#] A mock replaces a concrete class with an object that has the same methods and properties, but no behaviour by itself. Mocks can be configured by specifying return values, and one can make assertions about how they have been used. .. [#] Python does not have the notion of an interface like Java. Therefore an interface is implemented as an abstract base class (see also ``abc``). Method Interface ---------------- The Manager class provides the following public methods: .. function:: initialize() subscribe to events from the TcpServer and by starting the PLC uptime counter .. function:: shutdown() stops the TcpServer and properly shuts down the PLC uptime counter .. function:: start(port) starts a TCP server on the given port. If no port is given, the default 1234 is used. .. function:: stop() stops the TCP server .. function:: add_device(type, name) adds a soft device of the specified type with the specified name to the list of simluated soft devices, also injects exp. parameters for the device into the schema of BeckhoffSim .. function:: set_property(device_name, prop_name, value) sets a named PLC property on the specified device .. function:: set_io(device_name, io_name, value) sets a named hardware I/O property on the specified device Features -------- This section describes the features, provided by the Manager class, in more detail. Adding Devices ~~~~~~~~~~~~~~ Simulated soft devices are created using a factory. The factory method returns a device class depending on the specified type. The device type is a string that is also used in the related expected parameter of BeckhoffSim. The instance of the device class is stored into a dictionary using the device name as the key. Once the device is added to the dictionary, the Manager queries the list of properties and hardware I/Os from the device class instance. It then calls Framework methods to inject the properties and I/Os as expected parameters into the schema of BeckhoffSim. **Note**: PLC commands are intentionally not injected as expected parameters of BeckhoffSim. Networking ~~~~~~~~~~ When ``Manager.start()`` is called, the TCP server is started, and when ``Manager.stop()`` is called or if the Manager is destructed, the TCP server is stopped. When the TCP server is running, it fires events to indicate the occurence of a certain condition. The Manager therfore subscribes to the following events from the TCP server: error event is fired in case of errors on the TCP socket connect event is fired, when a client has connected to the TCP server disconnect event is fired, when a client has disconnected from the TCP server In case of a connect event, the Manager starts a thread to generate heartbeat messages each 10s. It also generates the self description message from it's set of soft devices, and passes it to the TCP server for sending the message to the network. In case of a disconnect event, the Manager stops the heartbeat thread. Time Source =========== The Time Source generates following time-dependent information that is used in the messages: - Time Stamp. Current time in seconds since the epoch as an integer value. - Time Fraction. Fractional part of the time as an integer with 100ns as a unit. - Train ID. A unique ID of the current XFEL shot. - Delta Time. The time difference between the start of message generation and arrival of a response from a device. An integer value with 100ns as a unit. Method Interface ---------------- A TimeSource class provides the following public methods: .. function:: initialize() starts a thread generating a new Train ID every 100ms .. function:: shutdown() stops the thread generating a new Train ID every 100ms .. function:: get_time_stamp() returns the Time Stamp .. function:: get_time_frac() returns the Time Fraction .. function:: get_train_id() returns the Train ID .. function:: generate_time_delta(): increases the current Time Delta by 1ms +- 0.1ms and returns this value .. function:: reset_time_delta() sets the current Time Delta to 0, the function should be called at the beginning of message generation TCP Server ========== The TCP Server opens a TCP socket and binds it to a port. It then starts to listen on that socket for client messages. If needed, it sends server messages on that socket. Messages are simply stream of bytes. There is no further processing of the message itself within the TCP server. The TCP server is implemented in the class ``TcpServer`` from module ``core.TcpServer``. It has no further dependencies on other classes. Method Interface ---------------- The TcpServer class provides the following public methods: .. function:: start(port) starts a TCP server on the given port .. function:: stop() stops the TCP server .. function:: send(message) sends a message over the network .. function:: subscribe_errorEvent(handler) subscribes to the error event by adding the specified handler .. function:: subscribe_connectEvent(handler) subscribes to the connect event by adding the specified handler .. function:: subscribe_disconnectEvent(handler) subscribes to the disconnect event by adding the specified handler Features -------- The features implemented within the TcpServer class are described in more detail in the following sections. Events ~~~~~~ The TcpServer defines a set of events, which are fired asynchronously: connect event is fired, when a client has connected to the TCP server disconnect event is fired, when a client has disconnected from the TCP server error event is fired in case of errors on the TCP socket Socket Thread ~~~~~~~~~~~~~ The socket related code is run in a separate thread. Data exchange with this thread is via 2 queues: an input queue for data received on the socket, and an output queue for data to be sent on the socket. The main loop of the thread listens on the socket for client data. The listen operation times out after 1ms to inspect the output queue. If the output queue is not empty, the messages therein are sent over the socket. Then the loop starts over again. The loop terminates, when ``TcpServer.stop()`` is called. This also terminates the socket thread. Receiving ~~~~~~~~~ When messages from connected clients are received, the receive event is fired containing the message as the event argument. Sending ~~~~~~~ If the ``TcpServer.send(...)`` method is called, the message is pushed onto the output queue, if a TCP connection has been established. If there is no TCP connection, then the error event is fired. Devices ======= Devices simulate the behaviour of PLC soft devices. They have the same properties than their PLC counterpart, but in addition they possibly also have hardware inputs and outputs. Properties and I/Os are annotated with meta information, such as name and value type. PLC properties and hardware I/Os are implemented as ``Property`` class from module ``devices.property``. The annotated meta information is a named tuple ``BeckhoffKey`` for PLC properties, and ``PropertySpec`` for I/O properties. Devices are implemented as classes derived from a base class ``BaseDevice`` from module ``devices.base``. For each PLC soft device, there is an associated simulated device. Currently, the following devices are supported: - ``DigitalInput`` Methods and Properties ---------------------- A device class provides the following public methods: .. function:: subscribe_update_property_event(handler) subscribes to the update event for PLC properties by adding the specified handler .. function:: subscribe_update_io_event(handler) subscribes to the update event for hardware I/Os by adding the specified handler .. function:: append_class_self_description(message) appends the class description to an existing message .. function:: set_property(name, value) sets the named PLC property to the specified value .. function:: get_property(name) returns the value of the named property .. function:: set_io(name, value) sets the named hardware I/O to the specified value .. function:: get_io(name) returns the value of the named hardware I/O A device class provides the following public properties: .. property:: device_id returns the device ID for a device instance .. property:: properties returns the dictionary with the PLC properties (key is property name, value is an instance of the Property class) .. property:: ios returns the dictionary with the hardware I/Os (key is io name, value is an instance of the Property class) Features -------- The features implemented within the device class are described in more detail in the following sections. PLC Properties ~~~~~~~~~~~~~~ The device provides a set of PLC properties and commands, which are identical to the ones in the corresponding PLC soft device. Each one has a unique name, which usually starts with capital 'C' for commands, and with capital 'A' for properties. The supported types are listed in the following table: ========= ================================ type description ========= ================================ tBOOL boolean tBYTE 8 bit unsigned integer tSINT 8 bit signed integer tWORD 16 bit unsigned integer tINT 16 bit signed integer tDWORD 32 bit unsigned integer tDINT 32 bit signed integer tULINT 64 bit unsigned integer tLINT 64 bit signed integer tREAL 32 bit floating point (float) tLREAL 64 bit floating point (double) tSTRING ASCII byte array tVOID command tMULTI array of 32 bit unsigned integer ========= ================================ The distinction between commands (something that can be executed) and properties (something that is associated with a value) is done solely based on whether the variable type is tVOID or not. The meta information for PLC commands and properties are stored as named tuple ``BeckhoffKey`` and contain name, id, type, access, unit, and prefix. PLC properties and commands are stored in a dictionary, where the name is the key and the value is an instance of a ``Property`` class. The ``Property`` class consists of the meta information and a place to store the value. The dictionary is accessable from outside via ``Property.properties``. The ``set_property()`` method is used to modify the value of a PLC property. The name, passed in as an argument, is used to look up in the ``self._properties`` dictionary to reference the ``Property`` instance in order to write to it's value. Additional specific device behaviour needs to be implemented afterwards. The ``get_property()`` method is used to return the actual value of a PLC property. The name, passed in as an argument, is used to look up in the ``self._properties`` dictionary to reference the ``Property`` instance in order to read it's value. Hardware I/O ~~~~~~~~~~~~ The device provides hardware inputs and outputs of the corresponding PLC soft device. Each one has a unique name, and can be of one of the types, as mentioned with PLC properties, except tSTRING, tVOID and tMULTI. The meta information for Hardware I/Os are stored as named tuple ``PropertySpec`` and contain name, and type. Hardware I/Os are stored in a dictionary, where the name is the key and the value is an instance of a ``Property`` class. The ``Property`` class consists of the meta information and a place to store the value. The dictionary is accessable from outside via ``Property.ios``. The ``set_io()`` method is used to modify the value of a hardware input. The name, passed in as an argument, is used to look up in the ``self._hw_ios`` dictionary to reference the ``Property`` instance in order to write to it's value. Additional specific device behaviour needs to be implemented afterwards. The ``get_io()`` method is used to return the actual value of a hardware input or output. The name, passed in as an argument, is used to look up in the ``self._hw_ios`` dictionary to reference the ``Property`` instance in order to read it's value. Messages ======== A message contains key value pairs prepended with a short header. The message header contains timestamp and train ID. Each key value pair consists of two numbers, one to identify the soft device instance, and one to identify the property or command on that instance, followed by 0 to 20 values. Before sending messages over the network, they need to be converted into byte streams. Received byte streams need to be converted back into messages. Messages are implemented in the ``Message`` and the ``Pair`` classes from module ``core.message``. Method Interface ---------------- The Pair class provides the following public methods: .. function:: add_uint32(pair) adds an integer value to the pair (maximal 32 bit) .. function:: add_string(pair) adds a string value to the pair (maximal 80 characters, including terminating '\0') .. function:: to_bytes() returns the pair as a byte stream The Message class provides the following public methods: .. function:: add_pair(pair) adds the specified pair to the message .. function:: to_bytes() returns the message as a byte stream Features -------- The features implemented within the Message and Pair classes are described in more detail in the following sections. Converting to Byte Stream ~~~~~~~~~~~~~~~~~~~~~~~~~ In order to send messages over the network, they must be converted into a stream of bytes. The format of the byte stream is defined by [FT2015]_. Conversion is handled by the ``Message.to_bytes()`` method, which returns a byte stream representation of the message. .. [FT2015] T. Freyermuth, J. Tolkiehn, "European XFEL PLC Karabo Interface Description," internal note IN-2015-09-30, European XFEL, 2015 FiFo ==== The FiFo class basically mocks the PLC behavior inside the beckhoffSim project. Totally, two FiFo instances are required to establish the communication. One TX fifo (transmitter) for the message parsing between the Manager and the TcpServer and an additional RX (receiver) fifo which is filled with messages coming from the TcpServer. The messages are subsequently dispatched in the Manager. Technically, the FiFo class uses queue objects for thread safe communication. The maxsize member variable has been set to infinity. Instead, for size handling a private member variable ``_max_bytecount`` is been set during initialization. The private member variable ``_current_bytecount`` is used to observe the current size of the queue. After each method the current bytecount is updated. Method Interface ---------------- The FiFo class provides the following public methods: .. function:: try_put(item) Safe method to put an item into the fifo. The bytecount of the item is checked before. .. function:: try_get() Safe method to return an item from the fifo. Only returns an item if the bytecount of the fifo is greater than zero. .. function:: put(item) Puts an item into the fifo. If the bytecount of the item together with the current bytecount of the fifo exceeds the max bytecount and ``OverFlowException`` is raised. .. function:: get() Returns an item from the fifo. If the current bytecount inside the fifo is equal to zero, an ``EmptyBufferException`` is raised. .. function:: clear() Atomically locks the fifo and clears the fifo list of items. .. function:: __len__() Overwritten built-in function to provide the current bytecount inside the fifo. BeckhoffSim =========== BeckhoffSim is the top level class. It has a lightweight implementation by simply passing down all method calls to the Manager. In addition, it implements the Framework interface, which is used by the Manager to interact with the Karabo framework, for example to inject expected parameters. State Machine ------------- The state machine of BeckhoffSim is fairly simple. It only has a ``Started``, ``Stopped`` and ``Error`` state. The state reflects the state of the TCP server. If the TCP server is running, then BeckhoffSim is in ``Started`` state, when the TCP server is not running, then BeckhoffSim is in ``Stopped`` state. If an error occurs during the start of the TCP server, or while the TCP server is running, BeckhoffSim changes into the ``Error`` state. Instantiation ------------- BeckhoffSim has an expected parameter to configure simulated soft devices. It can only be set before instantiation. Is is a ``TABLE_ELEMENT`` where the device type and instance name can be specified. When BeckhofSim is instantiated, it goes through the list of configured soft devices, calls ``Manager.add_device()`` for each of them, and then ``Manager.initialize()``. Finally, BeckhoffSim changes into the ``Stopped`` state, and the expected parameters for each device are injected below the ``Devices`` node[#]_. If however, an exception was thrown while calling one of the Manager methods, BeckhoffSim changes into the ``Error`` state. .. [#] Due to a bug in Karabo framework (see #10897), users need to activate ``Apply all`` once to have value updates on the newly injected parameters. Shutdown -------- When BeckhoffSim is shut down (or killed), it simply shuts down the Manager by calling ``Manager.shutdown()``. Reconfiguration --------------- If expected parameters of the configured soft devices are changed, the ``preReconfigure()`` hook is called. Within the hook, the changed device properties and I/Os are extracted and the manager methods ``set_io()`` and ``set_property()`` are called respectively. Commands -------- BeckhoffSim provides 2 commands: Start starts the TCP server at the configured port Stop stops the TCP server The port for the TCP server is configured using an expected parameter. It can be changed at runtime, but the TCP server needs to be restarted in order to take effect. Appendix ======== File Structure -------------- The code is organized into packages: core contains modules defining the manager, TCP Server, and classes associated with the manager devices contains modules to define types, enums, base device, and soft devices tests contains the unit tests Coding Style ------------ The BeckhoffSim code follows the guidelines as proposed by PEP 8 Style Guide available under PythonStyleGuide_. .. _PythonStyleGuide: https://www.python.org/dev/peps/pep-0008 Unit Tests ---------- Most part of BeckhoffSim is unit tested. Unit tests are also run as part of Continuous Integration (CI). In order to get test reporting on the CI server, XML runner is required. Since it is not yet part of the Karabo Python library[#]_, you need to install it into your user account by running .. [#] see bug report #10818 :: pip install --user unittest-xml-reporting Afterwards, it might be necessary to add the user site-packages directory to PYTHONPATH, which is :: ~/.local/lib/python3.4/site-packages under Linux, and :: %APPDATA%\Python\Python34\site-packages under Windows [#]_. .. [#] ``%APPDATA%`` resolves to ``\\win.desy.de\home\<user>\Application Data``.