{"id":2122,"date":"2021-12-16T14:30:14","date_gmt":"2021-12-16T13:30:14","guid":{"rendered":"https:\/\/weeeopen.polito.it\/?p=2122"},"modified":"2023-10-17T20:51:33","modified_gmt":"2023-10-17T18:51:33","slug":"implementazione-delle-librerie-twisted-per-protocolli-client-server-multithread-in-python","status":"publish","type":"post","link":"https:\/\/weeeopen.polito.it\/en\/implementazione-delle-librerie-twisted-per-protocolli-client-server-multithread-in-python\/","title":{"rendered":"Python Twisted library implementation for multithread client-server protocol"},"content":{"rendered":"<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p>During the development of <a href=\"https:\/\/github.com\/WEEE-Open\/pesto\">Pesto<\/a>, one of our software dedicated to data erase on hard disks\/SSDs, we had to decide which approach to use in managing the individual processes of erase or writing data to the devices. After a brief (almost non-existent) discussion, we chose a client-server model and then immediately set to work implementing the various functions.<\/p>\n\n\n\n<p>This choice was strongly influenced by the fact that it was convenient to perform disk operations remotely on another machine set up only to perform such actions.<\/p>\n\n\n\n<p>So we have divided program tasks into \"server tasks\" (read\/write operations on disks) and \"client tasks\" (receiving status\/results of operations and graphical interface).<\/p>\n\n\n\n<p>Having done that, we got stuck in the fundamental problem, that is the client-server communication protocol. Initially we developed a simple <strong>UDP <\/strong>protocol by using the <code>socket<\/code> libraries which, however, did not guarantee the complete reception of the commands by the server. So we switched to a <strong>TCP<\/strong> protocol, always using the <code>socket <\/code>libraries, but as the project progressed, small but serious protocol errors caused significant problems while complexity increased and thread management needed for the multiple operations the program must perform silently took over.<\/p>\n\n\n\n<p>All of this led us to discover and employ the <strong>Twisted<\/strong>libraries, saving us the trouble of having to write a protocol from scratch and allowing us to greatly speed up the writing of the other parts of the program.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What is Twisted and what is it for?<\/h2>\n\n\n\n<p>Twisted is an event-driven network programming framework that provides several useful tools for simplifying the writing of a network protocol.<\/p>\n\n\n\n<p>With Twisted you can build <em>echo servers<\/em>, <em>web servers<\/em>, <em>mail clients<\/em>, <em>SSH clients<\/em>, \u2026<\/p>\n\n\n\n<p>What we are interested in, in this case, is to build a client-server communication protocol able to allow the client to send specific commands following specific events caused by actions that the user performs on the graphical interface or on the terminal.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Server side implemetation<\/h2>\n\n\n\n<p>For illustration purposes, let's build a small <strong>TCP<\/strong> server capable of receiving text strings and processing them to perform specific actions.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">from twisted.internet import protocol, reactor\nfrom twisted.protocols.basic import LineOnlyReceiver\n\nclass Server(LineOnlyReceiver):\n    def connectionMade(self):\n        self.sendLine(b\"Connection with the server established!\")\n        print(\"Client connected.\")\n\n    def connectionLost(self, reason):\n        print(f\"Connection lost.\")\n\n    def dataReceived(self, data):\n        data = data.decode('utf-8').rstrip('\\n\\r')\n        print(f\"Received data: {data}\")\n        if data == 'hello':\n            self.sendLine(b\"Hello! How are you?\")\n        if data == 'goodbye':\n            self.sendLine(b\"Noooo don't go awayyyyy...\")\n            self.transport.loseConnection()\n        if 'write: ' in data:\n            data = data.lstrip(\"write: \")\n            if self.factory.saveText(data):\n                self.sendLine(b\"Message saved!\")\n\n\nclass ServerFactory(protocol.Factory):\n    protocol = Server\n\n    def startFactory(self):\n        self.fp = open('file.txt', 'a')\n\n    def stopFactory(self):\n        self.fp.close()\n\n    def saveText(self, data):\n        self.fp.write(data + '\\n')\n        self.fp.flush()\n        return True\n\n\ndef main():\n    reactor.listenTCP(1234, ServerFactory())\n    reactor.run()\n\n\nif __name__ == \"__main__\":\n    main()<\/pre>\n\n\n\n<p>There are 3 key elements in the server code:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Protocol \u2192 <code>Server<\/code><\/li><li>Factory \u2192 <code>ServerFactory<\/code><\/li><li>Reactor \u2192 <code>reactor<\/code><\/li><\/ul>\n\n\n\n<p>The <strong>reactor<\/strong> constitutes the twisted event loop and handles all internal program events and messages.<\/p>\n\n\n\n<p>The <strong>factory<\/strong> class is an object that allows us to separate the \"local\" functions from the communication functions that are left exclusively to the <strong>protocol<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Reactor<\/h4>\n\n\n\n<p class=\" translation-block\">The reactor is called in the <code>main()<\/code> function with the  <code>listenTCP<\/code> and <code>run<\/code> methods.<\/p>\n\n\n\n<p class=\" translation-block\">The<code>listenTCP(<em>porta<\/em>,<em> factory()<\/em>)<\/code><em> <\/em>command builds the actual <strong>server<\/strong>, using a specific <strong>port<\/strong> that the user enters as an argument and calling a <strong>factory<\/strong>, which will take care of creating the protocol and establishing the connection. In our case, we'll use the port <strong>1234<\/strong> and the factory <strong>ServerFactory<\/strong>.<\/p>\n\n\n\n<p class=\" translation-block\">With <code>run()<\/code>, the Twisted <strong>event loop<\/strong> is started and it will run until the server is terminated or a catastrophic error occurs.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Factory<\/h4>\n\n\n\n<p>A <strong>factory<\/strong> object can be seen as a <strong>protocol factory<\/strong> which also performs all the operation associated with a single connection.<\/p>\n\n\n\n<p>In this case we have used 2 \"prefabricated\" methods, <code>startFactory<\/code><strong> <\/strong>and <code>stopFactory<\/code>, and a method we wrote to save the text to a file, <code>saveText<\/code>.<\/p>\n\n\n\n<p class=\" translation-block\">The <code>startFactory(<em>self<\/em>)<\/code> method is called every time a new connection is requested and executes all the code inside it once. In our case, a <strong>file.txt<\/strong> file is opened in <em>append<\/em> mode.<\/p>\n\n\n\n<p class=\" translation-block\">The <code>stopFactory(<em>self<\/em>)<\/code><strong><\/strong> method is instead used when the specific connection is dropped and, as for<code>startFactory<\/code> it executes the code inside it only once. Here, it just closes the <strong>file.txt<\/strong> file.<\/p>\n\n\n\n<p>The <code>saveText(<em>self<\/em>, <em>data<\/em>)<\/code>method, written by us, is called by the protocol when a message contains the string '<em>write: <\/em>' inside it and writes to the <strong>file.txt<\/strong> file everything that follows the command.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Protocol<\/h4>\n\n\n\n<p>The <strong>protocol<\/strong> is the object that manages all the <strong>communication<\/strong> with the connected client.<\/p>\n\n\n\n<p>Three main methods are defined:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>connectionMade<\/code><\/li><li><code>connectionLost<\/code><\/li><li><code>dataReceived<\/code><\/li><\/ul>\n\n\n\n<p>The <code>connectionMade<\/code> method is called when a connection is established by executing the code inside it. In our case, a message is sent to the client confirming to the user that a connection has been made.<\/p>\n\n\n\n<p>The <code>connectionLost<\/code>method, as the name suggests, executes the code it contains when the connection is terminated or lost.<\/p>\n\n\n\n<p>The <code>dataReceived<\/code> method represents the core of the server, in a certain sense. In fact, inside this method it is established how the server should behave according to the messages it receives.<\/p>\n\n\n\n<p>When the connection is established, the server sends a confirmation message to the client so that the user can determine if the connection has indeed been established.<\/p>\n\n\n\n<p>If the client sends the command 'hello', the server will respond with 'Hello! How are you?<\/p>\n\n\n\n<p>If the client sends a '<em>write:<\/em>\u2019 command, where <em><strong>&lt;testo&gt;<\/strong><\/em> verr\u00e0 sostituito con un messaggio in particolare, il server salver\u00e0 il testo inviato in un file locale e risponder\u00e0 con \u2018<em>Testo salvato!<\/em>\u2019.<\/p>\n\n\n\n<p class=\" translation-block\">If the client sends the command \u2018<em>goodbye<\/em>\u2019, the server will respond with \u2018<em>Noooo don't go awayyyyy...<\/em>\u2019 and the connection will be closed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Client side implementation<\/h2>\n\n\n\n<p>Now that we have created our server, we need to communicate with it via a client. As in the case of the server, we have 3 key objects:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Protocol \u2192 <code>Client<\/code><\/li><li>Factory \u2192 <code>ClientFactory<\/code><\/li><li>Reactor Thread \u2192 <code>ReactorThread<\/code><\/li><\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">from twisted.protocols.basic import LineOnlyReceiver\nfrom twisted.internet import reactor, protocol\nfrom threading import Thread\n\nreceiver = None\n\n\nclass Client(LineOnlyReceiver):\n    def connectionMade(self):\n        global receiver\n        receiver = self\n\n    def connectionLost(self, reason):\n        print(\"Connection lost.\")\n\n    def lineReceived(self, line):\n        line = line.decode('utf-8')\n        self.factory.update(line)\n\n    def sendMsg(self, msg: str):\n        if self is None:\n            print(\"Cannot send message to server. No connection.\")\n        else:\n            msg = msg.encode('utf-8')\n            self.sendLine(msg)\n\n    def disconnect(self):\n        self.transport.loseConnection()\n\n\nclass ClientFactory(protocol.ClientFactory):\n    protocol = Client\n\n    def clientConnectionFailed(self, connector, reason):\n        print(\"Connection failed.\")\n\n    def update(self, data):\n        print(data)\n\n\nclass ReactorThread(Thread):\n    def __init__(self, host: str, port: int):\n        Thread.__init__(self)\n        self.host = host\n        self.port = port\n        self.protocol = Client\n        self.factory = ClientFactory()\n        self.reactor = reactor\n\n    def run(self):\n        self.reactor.connectTCP(self.host, self.port, self.factory)\n        self.reactor.run(installSignalHandlers=False)\n\n    def stop(self):\n        self.reactor.callFromThread(Client.disconnect, receiver)\n\n    def send(self, msg: str):\n        self.reactor.callFromThread(Client.sendMsg, receiver, msg)\n\n    def reconnect(self):\n        self.reactor.connectTCP(self.host, self.port, self.factory)\n        self.reactor.callFromThread(Client.disconnect, receiver)\n\ndef main():\n    host = \"127.0.0.1\"\n    port = 1234\n    r_thread = ReactorThread(host, port)\n    r_thread.start()\n    while True:\n        cmd = input()\n        if cmd == 'reconnect':\n            r_thread.reconnect()\n        else:\n            r_thread.send(cmd)\n\nif __name__ == '__main__':\n    main()<\/pre>\n\n\n\n<p class=\" translation-block\">In this program we will use the <code>threading<\/code> library  to separate client-server communication from the user interface execution. This is because if we start the <strong>reactor<\/strong> in the main thread, it \"takes over\" the process and the user will have to wait for the program to finish entering commands, which is not really ideal for our purposes.<\/p>\n\n\n\n<p class=\" translation-block\">So we have 2 threads per client: one thread (the <strong>main thread<\/strong>) to insert the commands to send to the server and another thread for the <strong>reactor<\/strong>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Reactor Thread<\/h4>\n\n\n\n<p>The reactor thread is defined in <code>ReactorThread<\/code> that is a Thread class which is instantiated at the beginning of the program.<\/p>\n\n\n\n<p class=\" translation-block\">When a new <code>ReactorThread<\/code> is instantiated is necessary to specify the <strong>address<\/strong> of the server to which you want to connect and the relative <strong>port<\/strong>. At thread startup the <strong>run<\/strong> method is executed and the reactor makes an attempt to connect to the server with the <code>connectTCP<\/code> method in which  must be specified the <strong>IP address<\/strong>, the <strong>port<\/strong> and the <strong>client factory<\/strong>.<\/p>\n\n\n\n<p>The other methods used in the example are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>stop<\/code> \u2192 Calls the protocol's <strong>disconnect<\/strong> method to make a disconnection.<\/li><li><code>send<\/code> \u2192 Calls the protocol's <strong>sendMsg <\/strong>method to send a message coming from the main thread.<\/li><li><code>reconnect<\/code> \u2192 Make a new connection attempt to the server (if the 'reconnect' command is sent from the main thread).<em>reconnect<\/em>&#8216;)<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Client Factory<\/h4>\n\n\n\n<p>As in the case of the server, here we have a client \"factory\" that can perform various local tasks leaving the protocol with the sole task of communicating with the server.<\/p>\n\n\n\n<p>In our example there are 2 methods:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>clientConnectionFailed<\/code> \u2192 Prints \u201c<em>Connection failed<\/em>\u201d if the client cannot establish a connection.<\/li><li><code>update<\/code><strong> <\/strong>\u2192 Print the message received from the server on the terminal.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Protocol<\/h4>\n\n\n\n<p>In the protocol we define the methods useful for communication with the server:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li class=\" translation-block\"><code>connectionMade<\/code> \u2192 it \u201cstores\u201d the active protocol in a global <code>receiver<\/code> variable usable in the <code>ReactorThread<\/code> so you can send commands written on the <strong>main thread<\/strong>.<\/li><li><code>connectionLost<\/code> \u2192 Prints \u201c<em>Connection lost<\/em>\" if the connection is broken by the server.<\/li><li class=\" translation-block\"><code>lineReceived<\/code> \u2192 It is called when a message arrives from the server and sends it to the <code>ClientFactory<\/code>'s <code>update<\/code> method to print the received message on the terminal.<\/li><li><code>sendMsg<\/code><strong> <\/strong>\u2192 If you are connected to the server it sends the message from the input of the <strong>main thread<\/strong>, otherwise it prints an error stating that there is no connection.<\/li><li><code>disconnect<\/code> \u2192 Breaks the connection with the server.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Final considerations<\/h2>\n\n\n\n<p>Although it would have been very interesting to build a \"low level\" communication protocol with the <code>socket<\/code>libraries, on the other hand we would have had to spend a lot of time and energy on this aspect, which in our case is not decisive because <a href=\"https:\/\/github.com\/WEEE-Open\/pesto\">Pesto<\/a> is not a project that requires such high efficiency in terms of communication between client and server.<\/p>\n\n\n\n<p>Therefore, <strong>Twisted<\/strong> has proven to be a powerful tool capable of greatly speeding up software development.<\/p>","protected":false},"excerpt":{"rendered":"<p>Introduzione Durante la scrittura del Pesto, uno dei nostri software dedicato alla completa epurazione dei dati su dischi\/SSD, abbiamo dovuto decidere quale approccio utilizzare nella gestione dei singoli processi di pulizia o scrittura di dati nei dispositivi. Dopo una breve (quasi inesistente) discussione, abbiamo scelto un modello client-server e quindi ci siamo messi subito al [&hellip;]<\/p>\n","protected":false},"author":15,"featured_media":2474,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"episode_type":"","audio_file":"","podmotor_file_id":"","podmotor_episode_id":"","cover_image":"","cover_image_id":"","duration":"","filesize":"","filesize_raw":"","date_recorded":"","explicit":"","block":"","ocean_post_layout":"","ocean_both_sidebars_style":"","ocean_both_sidebars_content_width":0,"ocean_both_sidebars_sidebars_width":0,"ocean_sidebar":"","ocean_second_sidebar":"","ocean_disable_margins":"enable","ocean_add_body_class":"","ocean_shortcode_before_top_bar":"","ocean_shortcode_after_top_bar":"","ocean_shortcode_before_header":"","ocean_shortcode_after_header":"","ocean_has_shortcode":"","ocean_shortcode_after_title":"","ocean_shortcode_before_footer_widgets":"","ocean_shortcode_after_footer_widgets":"","ocean_shortcode_before_footer_bottom":"","ocean_shortcode_after_footer_bottom":"","ocean_display_top_bar":"default","ocean_display_header":"default","ocean_header_style":"","ocean_center_header_left_menu":"","ocean_custom_header_template":"","ocean_custom_logo":0,"ocean_custom_retina_logo":0,"ocean_custom_logo_max_width":0,"ocean_custom_logo_tablet_max_width":0,"ocean_custom_logo_mobile_max_width":0,"ocean_custom_logo_max_height":0,"ocean_custom_logo_tablet_max_height":0,"ocean_custom_logo_mobile_max_height":0,"ocean_header_custom_menu":"","ocean_menu_typo_font_family":"","ocean_menu_typo_font_subset":"","ocean_menu_typo_font_size":0,"ocean_menu_typo_font_size_tablet":0,"ocean_menu_typo_font_size_mobile":0,"ocean_menu_typo_font_size_unit":"px","ocean_menu_typo_font_weight":"","ocean_menu_typo_font_weight_tablet":"","ocean_menu_typo_font_weight_mobile":"","ocean_menu_typo_transform":"","ocean_menu_typo_transform_tablet":"","ocean_menu_typo_transform_mobile":"","ocean_menu_typo_line_height":0,"ocean_menu_typo_line_height_tablet":0,"ocean_menu_typo_line_height_mobile":0,"ocean_menu_typo_line_height_unit":"","ocean_menu_typo_spacing":0,"ocean_menu_typo_spacing_tablet":0,"ocean_menu_typo_spacing_mobile":0,"ocean_menu_typo_spacing_unit":"","ocean_menu_link_color":"","ocean_menu_link_color_hover":"","ocean_menu_link_color_active":"","ocean_menu_link_background":"","ocean_menu_link_hover_background":"","ocean_menu_link_active_background":"","ocean_menu_social_links_bg":"","ocean_menu_social_hover_links_bg":"","ocean_menu_social_links_color":"","ocean_menu_social_hover_links_color":"","ocean_disable_title":"default","ocean_disable_heading":"default","ocean_post_title":"","ocean_post_subheading":"","ocean_post_title_style":"","ocean_post_title_background_color":"","ocean_post_title_background":0,"ocean_post_title_bg_image_position":"","ocean_post_title_bg_image_attachment":"","ocean_post_title_bg_image_repeat":"","ocean_post_title_bg_image_size":"","ocean_post_title_height":0,"ocean_post_title_bg_overlay":0.5,"ocean_post_title_bg_overlay_color":"","ocean_disable_breadcrumbs":"default","ocean_breadcrumbs_color":"","ocean_breadcrumbs_separator_color":"","ocean_breadcrumbs_links_color":"","ocean_breadcrumbs_links_hover_color":"","ocean_display_footer_widgets":"default","ocean_display_footer_bottom":"default","ocean_custom_footer_template":"","ocean_post_oembed":"","ocean_post_self_hosted_media":"","ocean_post_video_embed":"","ocean_link_format":"","ocean_link_format_target":"self","ocean_quote_format":"","ocean_quote_format_link":"post","ocean_gallery_link_images":"on","ocean_gallery_id":[],"footnotes":""},"categories":[1],"tags":[23,22],"ppma_author":[170],"class_list":["post-2122","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","tag-divulgazione","tag-progetti_software","entry","has-media"],"authors":[{"term_id":170,"user_id":15,"is_guest":0,"slug":"palmi","display_name":"Niccol\u00f2 Palmieri","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/a013c919519414c1b75f9f42f9bfe72e93a234e7e8b149c93038e42ca83bca3f?s=96&d=retro&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/posts\/2122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/comments?post=2122"}],"version-history":[{"count":1,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/posts\/2122\/revisions"}],"predecessor-version":[{"id":3632,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/posts\/2122\/revisions\/3632"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/media\/2474"}],"wp:attachment":[{"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/media?parent=2122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/categories?post=2122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/tags?post=2122"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/weeeopen.polito.it\/en\/wp-json\/wp\/v2\/ppma_author?post=2122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}